[jira] [Updated] (SPARK-25222) Spark on Kubernetes Pod Watcher dumps raw container status

2018-11-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-25222: --- Fix Version/s: 3.0.0 > Spark on Kubernetes Pod Watcher dumps raw container sta

[jira] [Assigned] (SPARK-25222) Spark on Kubernetes Pod Watcher dumps raw container status

2018-11-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25222: -- Assignee: Rob Vesse > Spark on Kubernetes Pod Watcher dumps raw container sta

Re: Test and support only LTS JDK release?

2018-11-06 Thread Marcelo Vanzin
+1, that's always been my view. Although, to be fair, and as Sean mentioned, the jump from jdk8 is probably the harder part. After that it's less likely (hopefully?) that we'll run into issues in non-LTS releases. And even if we don't officially support them, trying to keep up with breaking change

[jira] [Resolved] (SPARK-25871) Streaming WAL should not use hdfs erasure coding, regardless of FS defaults

2018-11-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25871. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22882 [https

[jira] [Assigned] (SPARK-25871) Streaming WAL should not use hdfs erasure coding, regardless of FS defaults

2018-11-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25871: -- Assignee: Imran Rashid > Streaming WAL should not use hdfs erasure cod

[jira] [Assigned] (LIVY-525) Response Content-Type is application/json, but a simple string gets returned

2018-11-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-525: --- Assignee: Ingo Schuster > Response Content-Type is application/json, but a simple string g

[jira] [Resolved] (LIVY-525) Response Content-Type is application/json, but a simple string gets returned

2018-11-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-525. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 127 [https

[jira] [Commented] (SPARK-25590) kubernetes-model-2.0.0.jar masks default Spark logging config

2018-11-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673689#comment-16673689 ] Marcelo Vanzin commented on SPARK-25590: {noformat} $ jar tf assembly/ta

[jira] [Assigned] (SPARK-25827) Replicating a block > 2gb with encryption fails

2018-11-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25827: -- Assignee: Imran Rashid > Replicating a block > 2gb with encryption

[jira] [Resolved] (SPARK-25827) Replicating a block > 2gb with encryption fails

2018-11-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25827. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved by

[jira] [Commented] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673431#comment-16673431 ] Marcelo Vanzin commented on SPARK-25924: [~shaneknapp], it seems that the

[jira] [Created] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-01 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25924: -- Summary: k8s integration tests on jenkins do not save log output Key: SPARK-25924 URL: https://issues.apache.org/jira/browse/SPARK-25924 Project: Spark

[jira] [Created] (SPARK-25920) Avoid custom processing of CLI options for cluster submission

2018-11-01 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25920: -- Summary: Avoid custom processing of CLI options for cluster submission Key: SPARK-25920 URL: https://issues.apache.org/jira/browse/SPARK-25920 Project: Spark

[jira] [Resolved] (SPARK-25912) Starting history server throwing exception, when enable the local store directory

2018-11-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25912. Resolution: Not A Problem > Starting history server throwing exception, when enable

[jira] [Commented] (SPARK-25912) Starting history server throwing exception, when enable the local store directory

2018-11-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672043#comment-16672043 ] Marcelo Vanzin commented on SPARK-25912: No. As I said, that is how directo

[jira] [Commented] (SPARK-25912) Starting history server throwing exception, when enable the local store directory

2018-11-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672006#comment-16672006 ] Marcelo Vanzin commented on SPARK-25912: That's how directories wo

[jira] [Commented] (SPARK-25912) Starting history server throwing exception, when enable the local store directory

2018-11-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671966#comment-16671966 ] Marcelo Vanzin commented on SPARK-25912: Sorry, what's the bug

[jira] [Assigned] (SPARK-25837) Web UI does not respect spark.ui.retainedJobs in some instances

2018-11-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25837: -- Assignee: Patrick Brown (was: Apache Spark) > Web UI does not resp

[jira] [Resolved] (SPARK-25837) Web UI does not respect spark.ui.retainedJobs in some instances

2018-11-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25837. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0

[jira] [Commented] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2018-11-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671828#comment-16671828 ] Marcelo Vanzin commented on SPARK-20608: That error doesn't provide

[jira] [Created] (SPARK-25903) Flaky test: BarrierTaskContextSuite.throw exception on barrier() call timeout

2018-10-31 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25903: -- Summary: Flaky test: BarrierTaskContextSuite.throw exception on barrier() call timeout Key: SPARK-25903 URL: https://issues.apache.org/jira/browse/SPARK-25903

Re: [VOTE] SPARK 2.4.0 (RC5)

2018-10-31 Thread Marcelo Vanzin
+1 On Mon, Oct 29, 2018 at 3:22 AM Wenchen Fan wrote: > > Please vote on releasing the following candidate as Apache Spark version > 2.4.0. > > The vote is open until November 1 PST and passes if a majority +1 PMC votes > are cast, with > a minimum of 3 +1 votes. > > [ ] +1 Release this package

[jira] [Resolved] (SPARK-25855) Don't use Erasure Coding for event log files

2018-10-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25855. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22881 [https

[jira] [Assigned] (SPARK-25855) Don't use Erasure Coding for event log files

2018-10-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25855: -- Assignee: Imran Rashid > Don't use Erasure Coding for event l

[jira] [Created] (SPARK-25897) Cannot run k8s integration tests in sbt

2018-10-31 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25897: -- Summary: Cannot run k8s integration tests in sbt Key: SPARK-25897 URL: https://issues.apache.org/jira/browse/SPARK-25897 Project: Spark Issue Type

[jira] [Resolved] (SPARK-24746) AWS S3 301 Moved Permanently error message even after setting fs.s3a.endpoint for bucket in Mumbai region.

2018-10-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24746. Resolution: Information Provided In general, please use the mailing list for questions

[jira] [Commented] (HDFS-14038) Expose HdfsDataOutputStreamBuilder to include Spark in LimitedPrivate

2018-10-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669087#comment-16669087 ] Marcelo Vanzin commented on HDFS-14038: --- No, the problem here is not the us

[jira] [Created] (SPARK-25878) Document existing k8s features and how to add new features

2018-10-29 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25878: -- Summary: Document existing k8s features and how to add new features Key: SPARK-25878 URL: https://issues.apache.org/jira/browse/SPARK-25878 Project: Spark

[jira] [Created] (SPARK-25877) Put all feature-related code in the feature step itself

2018-10-29 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25877: -- Summary: Put all feature-related code in the feature step itself Key: SPARK-25877 URL: https://issues.apache.org/jira/browse/SPARK-25877 Project: Spark

[jira] [Created] (SPARK-25876) Simplify configuration types in k8s backend

2018-10-29 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25876: -- Summary: Simplify configuration types in k8s backend Key: SPARK-25876 URL: https://issues.apache.org/jira/browse/SPARK-25876 Project: Spark Issue Type

[jira] [Created] (SPARK-25875) Merge code to set up driver features for different languages

2018-10-29 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25875: -- Summary: Merge code to set up driver features for different languages Key: SPARK-25875 URL: https://issues.apache.org/jira/browse/SPARK-25875 Project: Spark

[jira] [Updated] (SPARK-25874) Simplify abstractions in the K8S backend

2018-10-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-25874: --- Issue Type: Umbrella (was: Improvement) > Simplify abstractions in the K8S back

[jira] [Created] (SPARK-25874) Simplify abstractions in the K8S backend

2018-10-29 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25874: -- Summary: Simplify abstractions in the K8S backend Key: SPARK-25874 URL: https://issues.apache.org/jira/browse/SPARK-25874 Project: Spark Issue Type

[jira] [Created] (SPARK-25857) Document delegation token code in Spark

2018-10-26 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25857: -- Summary: Document delegation token code in Spark Key: SPARK-25857 URL: https://issues.apache.org/jira/browse/SPARK-25857 Project: Spark Issue Type

[jira] [Resolved] (SPARK-24787) Events being dropped at an alarming rate due to hsync being slow for eventLogging

2018-10-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24787. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved by

[jira] [Assigned] (SPARK-24787) Events being dropped at an alarming rate due to hsync being slow for eventLogging

2018-10-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24787: -- Assignee: Devaraj K > Events being dropped at an alarming rate due to hsync be

[jira] [Resolved] (SPARK-25803) The -n option to docker-image-tool.sh causes other options to be ignored

2018-10-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25803. Resolution: Fixed Fix Version/s: 3.0.0 2.4.1 > The -n option

[jira] [Assigned] (SPARK-25803) The -n option to docker-image-tool.sh causes other options to be ignored

2018-10-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25803: -- Assignee: Steve Larkin > The -n option to docker-image-tool.sh causes other opti

[jira] [Assigned] (LIVY-495) Add basic UI for thriftserver

2018-10-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-495: --- Assignee: Marco Gaido > Add basic UI for thriftser

[jira] [Assigned] (LIVY-531) Update spark 2.3 default version to 2.3.2

2018-10-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-531: --- Assignee: Lantao Jin > Update spark 2.3 default version to 2.

[jira] [Resolved] (LIVY-495) Add basic UI for thriftserver

2018-10-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-495. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 114 [https

[jira] [Resolved] (LIVY-531) Update spark 2.3 default version to 2.3.2

2018-10-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-531. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 124 [https

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

2018-10-25 Thread Marcelo Vanzin
as my production application continues to submit jobs every once in a while, > the issue persists. > > On Wed, Oct 24, 2018 at 5:05 PM Marcelo Vanzin wrote: >> >> When you say many jobs at once, what ballpark are you talking about? >> >> The code in 2.3+ does try

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

2018-10-24 Thread Marcelo Vanzin
; import scala.concurrent._ > scala> import scala.concurrent.ExecutionContext.Implicits.global > scala> for (i <- 0 until 5) { Future { println(sc.parallelize(0 until > i).collect.length) } } > > On Mon, Oct 22, 2018 at 11:25 AM Marcelo Vanzin wrote: >> >> Just tried on 2.3.2 and wor

[jira] [Commented] (SPARK-25815) Kerberos Support in Kubernetes resource manager (Client Mode)

2018-10-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662761#comment-16662761 ] Marcelo Vanzin commented on SPARK-25815: I actually have this working (and

[jira] [Commented] (SPARK-25814) spark driver runs out of memory on org.apache.spark.util.kvstore.InMemoryStore

2018-10-23 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661269#comment-16661269 ] Marcelo Vanzin commented on SPARK-25814: That's UI data. You can co

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

2018-10-22 Thread Marcelo Vanzin
Just tried on 2.3.2 and worked fine for me. UI had a single job and a single stage (+ the tasks related to that single stage), same thing in memory (checked with jvisualvm). On Sat, Oct 20, 2018 at 6:45 PM Marcelo Vanzin wrote: > > On Tue, Oct 16, 2018 at 9:34 AM Patrick Brown > wro

[jira] [Commented] (SPARK-25783) Spark shell fails because of jline incompatibility

2018-10-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659129#comment-16659129 ] Marcelo Vanzin commented on SPARK-25783: Hmm... if the jline classes are in

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

2018-10-20 Thread Marcelo Vanzin
On Tue, Oct 16, 2018 at 9:34 AM Patrick Brown wrote: > I recently upgraded to spark 2.3.1 I have had these same settings in my spark > submit script, which worked on 2.0.2, and according to the documentation > appear to not have changed: > > spark.ui.retainedTasks=1 > spark.ui.retainedStages=1 >

[jira] [Commented] (SPARK-13478) Fetching delegation tokens for Hive fails when using proxy users

2018-10-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16658025#comment-16658025 ] Marcelo Vanzin commented on SPARK-13478: If you run as user1 with

[jira] [Commented] (SPARK-25783) Spark shell fails because of jline incompatibility

2018-10-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657520#comment-16657520 ] Marcelo Vanzin commented on SPARK-25783: In fact, the dependency used to e

[jira] [Assigned] (SPARK-25745) docker-image-tool.sh ignores errors from Docker

2018-10-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25745: -- Assignee: Rob Vesse > docker-image-tool.sh ignores errors from Doc

[jira] [Resolved] (SPARK-25745) docker-image-tool.sh ignores errors from Docker

2018-10-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25745. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22748 [https

[jira] [Commented] (SPARK-25783) Spark shell fails because of jline incompatibility

2018-10-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657435#comment-16657435 ] Marcelo Vanzin commented on SPARK-25783: I think the problem that the &quo

[jira] [Commented] (SPARK-13478) Fetching delegation tokens for Hive fails when using proxy users

2018-10-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657029#comment-16657029 ] Marcelo Vanzin commented on SPARK-13478: If you have a principal and a ke

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2018-10-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657028#comment-16657028 ] Marcelo Vanzin commented on SPARK-19143: I don't think there&#

[jira] [Assigned] (SPARK-25683) Updated the log for the firstTime event Drop occurs.

2018-10-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25683: -- Assignee: Shivu Sondur > Updated the log for the firstTime event Drop occ

[jira] [Resolved] (SPARK-25683) Updated the log for the firstTime event Drop occurs.

2018-10-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25683. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22677 [https

[jira] [Resolved] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25682. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22681 [https

[jira] [Assigned] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25682: -- Assignee: Marcelo Vanzin > Docker images generated from dev build and from d

[jira] [Commented] (SPARK-25766) AMCredentialRenewer can leak FS clients

2018-10-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655574#comment-16655574 ] Marcelo Vanzin commented on SPARK-25766: That code doesn't exist anymo

Re: Starting to make changes for Spark 3 -- what can we delete?

2018-10-16 Thread Marcelo Vanzin
Might be good to take a look at things marked "@DeveloperApi" and whether they should stay that way. e.g. I was looking at SparkHadoopUtil and I've always wanted to just make it private to Spark. I don't see why apps would need any of those methods. On Tue, Oct 16, 2018 at 10:18 AM Sean Owen wrot

[jira] [Resolved] (SPARK-25693) Fix the multiple "manager" classes in org.apache.spark.deploy.security

2018-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25693. Resolution: Duplicate I started looking at this while waiting for reviews of SPARK-23781

[jira] [Assigned] (SPARK-25394) Expose App status metrics as Source

2018-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25394: -- Assignee: Stavros Kontopoulos > Expose App status metrics as Sou

[jira] [Resolved] (SPARK-25394) Expose App status metrics as Source

2018-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25394. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22381 [https

[jira] [Assigned] (LIVY-529) Fix supported Python version from 2.6+ to 2.7+ in documentation

2018-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-529: --- Assignee: leehaejoon > Fix supported Python version from 2.6+ to 2.7+ in documentat

[jira] [Resolved] (LIVY-529) Fix supported Python version from 2.6+ to 2.7+ in documentation

2018-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-529. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 122 [https

[jira] [Commented] (SPARK-25748) Upgrade net.razorvine:pyrolite version

2018-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652109#comment-16652109 ] Marcelo Vanzin commented on SPARK-25748: That CVE seems completely unrelate

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652029#comment-16652029 ] Marcelo Vanzin commented on SPARK-25732: bq. livy server would not know w

[jira] [Resolved] (SPARK-25738) LOAD DATA INPATH doesn't work if hdfs conf includes port

2018-10-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25738. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22733 [https

[jira] [Assigned] (SPARK-25738) LOAD DATA INPATH doesn't work if hdfs conf includes port

2018-10-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25738: -- Assignee: Imran Rashid > LOAD DATA INPATH doesn't work if hdfs conf inclu

[jira] [Assigned] (SPARK-23257) Implement Kerberos Support in Kubernetes resource manager

2018-10-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23257: -- Assignee: Ilan Filonenko > Implement Kerberos Support in Kubernetes resource mana

[jira] [Resolved] (SPARK-23257) Implement Kerberos Support in Kubernetes resource manager

2018-10-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23257. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21669 [https

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650591#comment-16650591 ] Marcelo Vanzin commented on SPARK-25732: In fact, do you even need proxy

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650535#comment-16650535 ] Marcelo Vanzin commented on SPARK-25732: I'd have preferred a system w

[jira] [Commented] (SPARK-13478) Fetching delegation tokens for Hive fails when using proxy users

2018-10-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650451#comment-16650451 ] Marcelo Vanzin commented on SPARK-13478: You don't need a keytab to

Re: kerberos auth for MS SQL server jdbc driver

2018-10-15 Thread Marcelo Vanzin
Spark only does Kerberos authentication on the driver. For executors it currently only supports Hadoop's delegation tokens for Kerberos. To use something that does not support delegation tokens you have to manually manage the Kerberos login in your code that runs in executors, which might be trick

[jira] [Resolved] (SPARK-20327) Add CLI support for YARN custom resources, like GPUs

2018-10-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-20327. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 20761 [https

[jira] [Assigned] (SPARK-20327) Add CLI support for YARN custom resources, like GPUs

2018-10-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-20327: -- Assignee: Szilard Nemeth > Add CLI support for YARN custom resources, like G

[jira] [Updated] (SPARK-25636) spark-submit swallows the failure reason when there is an error connecting to master

2018-10-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-25636: --- Fix Version/s: (was: 2.4.0) 3.0.0 2.4.1 > sp

Re: Remove Flume support in 3.0.0?

2018-10-10 Thread Marcelo Vanzin
BTW, although I did not file a bug for that, I think we should also consider getting rid of the kafka-0.8 connector. That would leave only kafka-0.10 as the single remaining dstream connector in Spark, though. (If you ignore kinesis which we can't ship in binary form or something like that?) On We

Re: moving the spark jenkins job builder repo from dbricks --> spark

2018-10-10 Thread Marcelo Vanzin
Thanks for doing this. The more things we have accessible to the project members in general the better! (Now there's that hive fork repo somewhere, but let's not talk about that.) On Wed, Oct 10, 2018 at 9:30 AM shane knapp wrote: >> > * the JJB templates are able to be run by anyone w/jenkins l

[jira] [Assigned] (SPARK-25636) spark-submit swallows the failure reason when there is an error connecting to master

2018-10-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25636: -- Assignee: Devaraj K > spark-submit swallows the failure reason when there is

[jira] [Resolved] (SPARK-25636) spark-submit swallows the failure reason when there is an error connecting to master

2018-10-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25636. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22623 [https

[jira] [Commented] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644194#comment-16644194 ] Marcelo Vanzin commented on SPARK-25682: I was busy with other things but I

[jira] [Created] (SPARK-25693) Fix the multiple "manager" classes in org.apache.spark.deploy.security

2018-10-09 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25693: -- Summary: Fix the multiple "manager" classes in org.apache.spark.deploy.security Key: SPARK-25693 URL: https://issues.apache.org/jira/browse/SPARK-25693

[jira] [Created] (SPARK-25689) Move token renewal logic to driver in yarn-client mode

2018-10-09 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25689: -- Summary: Move token renewal logic to driver in yarn-client mode Key: SPARK-25689 URL: https://issues.apache.org/jira/browse/SPARK-25689 Project: Spark

[jira] [Resolved] (LIVY-524) Basic Auth support in curl requests to LIVY REST API

2018-10-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-524. - Resolution: Duplicate Please search JIRA before filing a new feature request or bug. > Basic A

Re: GitHub emails on dev

2018-10-08 Thread Marcelo Vanzin
Maybe infra did something to re-enable it. We asked them to disable it way back (INFRA-14973). Maybe reopen that ticket or file a new one. On Mon, Oct 8, 2018 at 4:19 PM Alex Bozarth wrote: > > > > Is there a reason we've begun getting GitHub PR emails on the dev list? The > regular glut of emails

[jira] [Created] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-08 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25682: -- Summary: Docker images generated from dev build and from dist tarball are different Key: SPARK-25682 URL: https://issues.apache.org/jira/browse/SPARK-25682

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

2018-10-08 Thread Marcelo Vanzin
On Mon, Oct 8, 2018 at 6:36 AM Rob Vesse wrote: > Since connectivity back to the client is a potential stumbling block for > cluster mode I wander if it would be better to think in reverse i.e. rather > than having the driver pull from the client have the client push to the > driver pod? > > Yo

[jira] [Resolved] (LIVY-516) Secure Livy Rest API

2018-10-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-516. - Resolution: Information Provided Fix Version/s: (was: 0.5.0) See LIVY-508. > Sec

[jira] [Resolved] (LIVY-514) No LivyClientFactory implementation was found

2018-10-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-514. - Resolution: Cannot Reproduce Without an answer to the above I can only say it's a problem on

[jira] [Resolved] (LIVY-523) Support Spark 2.4

2018-10-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-523. - Resolution: Duplicate Oops forgot I already filed it. > Support Spark

[jira] [Created] (LIVY-523) Support Spark 2.4

2018-10-05 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created LIVY-523: --- Summary: Support Spark 2.4 Key: LIVY-523 URL: https://issues.apache.org/jira/browse/LIVY-523 Project: Livy Issue Type: Task Components: Core

Re: Enable HTTP Basic Authenticaton for Livy Rest APIs

2018-10-05 Thread Marcelo Vanzin
ilor our Enterprise > requirements. > > Is there any developments guidelined available to get me started to change > the code? > > On Fri, Oct 5, 2018, 2:50 PM Marcelo Vanzin wrote: > >> I don't think Livy currently has support for basic auth. >> >> Although this PR

[jira] [Resolved] (LIVY-500) Add client for testing thriftserver

2018-10-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-500. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 113 [https

[jira] [Assigned] (LIVY-500) Add client for testing thriftserver

2018-10-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-500: --- Assignee: Marco Gaido > Add client for testing thriftser

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

2018-10-05 Thread Marcelo Vanzin
On Fri, Oct 5, 2018 at 7:54 AM Rob Vesse wrote: > Ideally this would all just be handled automatically for users in the way > that all other resource managers do I think you're giving other resource managers too much credit. In cluster mode, only YARN really distributes local dependencies, becau

[jira] [Resolved] (CRYPTO-141) Errors in native code can leave Java wrappers in bad state

2018-10-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/CRYPTO-141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved CRYPTO-141. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 1.1.0 > Errors

<    4   5   6   7   8   9   10   11   12   13   >