[jira] [Assigned] (LIVY-551) Add "doAs" impersonation support

2019-02-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-551: --- Assignee: Kevin Risden > Add "doAs" imperson

[jira] [Resolved] (LIVY-551) Add "doAs" impersonation support

2019-02-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-551. - Resolution: Fixed Issue resolved by pull request 141 [https://github.com/apache/incubator-livy

[incubator-livy] branch master updated: [LIVY-551] Add "doAs" impersonation support

2019-02-07 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-livy.git The following commit(s) were added to refs/heads/master by this push: new 80daade [LIVY-551] Add "

[jira] [Resolved] (SPARK-26842) java.lang.IllegalArgumentException: Unsupported class file major version 55

2019-02-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26842. Resolution: Invalid Java 11 is not supported in 2.4

[jira] [Resolved] (SPARK-26733) Clean up entrypoint.sh

2019-02-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26733. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23655 [https

[jira] [Assigned] (SPARK-26733) Clean up entrypoint.sh

2019-02-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26733: -- Assignee: Marcelo Vanzin > Clean up entrypoint

[spark] branch master updated: [SPARK-26733][K8S] Cleanup entrypoint.sh.

2019-02-05 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c624f5d [SPARK-26733][K8S] Cleanup

Re: Livy-0.6 release?

2019-02-05 Thread Marcelo Vanzin
4.. > > Marcelo, what particular patches are blocking Livy 0.6 release? > > I see 3 jiras with 0.6 as Fix Version - not sure if that's correct way to > find blockers. > https://goo.gl/9axfsw > > > Thank you! > Ruslan Dautkhanov > > > On Mon, Jan 28, 2019 at 2

[jira] [Resolved] (LIVY-41) Need a way to submit codes through Named Session instead of Session ID numbers

2019-02-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-41. Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 48 [https://github.com

[jira] [Assigned] (LIVY-41) Need a way to submit codes through Named Session instead of Session ID numbers

2019-02-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-41: -- Assignee: Meisam > Need a way to submit codes through Named Session instead of Session

[incubator-livy] branch master updated: [LIVY-41] Let users access sessions by session name

2019-02-05 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-livy.git The following commit(s) were added to refs/heads/master by this push: new def6318 [LIVY-41] Let users access

[jira] [Assigned] (SPARK-10614) SystemClock uses non-monotonic time in its wait logic

2019-02-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-10614: -- Assignee: (was: Marcelo Vanzin) > SystemClock uses non-monotonic time in

[jira] [Assigned] (SPARK-10614) SystemClock uses non-monotonic time in its wait logic

2019-02-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-10614: -- Assignee: Marcelo Vanzin > SystemClock uses non-monotonic time in its wait lo

[jira] [Issue Comment Deleted] (SPARK-24432) Add support for dynamic resource allocation

2019-01-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24432: --- Comment: was deleted (was: vanzin closed pull request #22722: [SPARK-24432][k8s] Add

[jira] [Assigned] (SPARK-24432) Add support for dynamic resource allocation

2019-01-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24432: -- Assignee: Marcelo Vanzin > Add support for dynamic resource allocat

[jira] [Assigned] (SPARK-24432) Add support for dynamic resource allocation

2019-01-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24432: -- Assignee: (was: Marcelo Vanzin) > Add support for dynamic resource allocat

[jira] [Updated] (SPARK-26726) Synchronize the amount of memory used by the broadcast variable to the UI display

2019-01-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26726: --- Fix Version/s: 2.3.3 > Synchronize the amount of memory used by the broadcast variable

[jira] [Assigned] (SPARK-26726) Synchronize the amount of memory used by the broadcast variable to the UI display

2019-01-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26726: -- Assignee: hantiantian > Synchronize the amount of memory used by the broadc

[spark] branch branch-2.3 updated: [SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

2019-01-31 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new 94a4b46 [SPARK-26726] Synchronize the

[jira] [Resolved] (SPARK-26726) Synchronize the amount of memory used by the broadcast variable to the UI display

2019-01-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26726. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved by

[spark] branch branch-2.4 updated: [SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

2019-01-31 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new d9403e4 [SPARK-26726] Synchronize the

[spark] branch master updated: [SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

2019-01-31 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f4a17e9 [SPARK-26726] Synchronize the amount of

[jira] [Commented] (SPARK-23155) YARN-aggregated executor/driver logs appear unavailable when NM is down

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16756602#comment-16756602 ] Marcelo Vanzin commented on SPARK-23155: For those looking the patc

[jira] [Resolved] (SPARK-23155) YARN-aggregated executor/driver logs appear unavailable when NM is down

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23155. Resolution: Fixed Assignee: Jungtaek Lim Fix Version/s: 3.0.0 > Y

[jira] [Resolved] (SPARK-26784) Allow running driver pod as provided user

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26784. Resolution: Won't Fix As far as I know you can do that with pod templates. So no poi

[jira] [Created] (SPARK-26788) Remove SchedulerExtensionService

2019-01-30 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26788: -- Summary: Remove SchedulerExtensionService Key: SPARK-26788 URL: https://issues.apache.org/jira/browse/SPARK-26788 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-26737) Executor/Task STDERR & STDOUT log urls are not correct in Yarn deployment mode

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26737: --- Fix Version/s: 3.0.0 > Executor/Task STDERR & STDOUT log urls are not correct

[jira] [Resolved] (SPARK-26737) Executor/Task STDERR & STDOUT log urls are not correct in Yarn deployment mode

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26737. Resolution: Fixed Assignee: Jungtaek Lim The patch for SPARK-26311 ended up fixing

[jira] [Assigned] (SPARK-26311) [YARN] New feature: custom log URL for stdout/stderr

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26311: -- Assignee: Jungtaek Lim > [YARN] New feature: custom log URL for stdout/std

[spark] branch master updated: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS

2019-01-30 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new ae5b2a6 [SPARK-26311][CORE] New feature: apply

[jira] [Resolved] (SPARK-26753) Log4j customization not working for spark-shell

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26753. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23675 [https

[jira] [Assigned] (SPARK-26753) Log4j customization not working for spark-shell

2019-01-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26753: -- Assignee: Ankur Gupta > Log4j customization not working for spark-sh

[spark] branch master updated: [SPARK-26753][CORE] Fixed custom log levels for spark-shell by using Filter instead of Threshold

2019-01-30 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 25b97a4 [SPARK-26753][CORE] Fixed custom log

[spark] branch master updated: [SPARK-25689][CORE] Follow up: don't get delegation tokens when kerberos not available.

2019-01-30 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a8da410 [SPARK-25689][CORE] Follow up: don&#

[jira] [Commented] (SPARK-26766) Remove the list of filesystems from HadoopDelegationTokenProvider.obtainDelegationTokens

2019-01-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755501#comment-16755501 ] Marcelo Vanzin commented on SPARK-26766: The only thing YARN-specific a

[spark-website] branch asf-site updated: CVE-2018-11760 (#175)

2019-01-28 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new d73a92d CVE-2018-11760 (#175

[jira] [Assigned] (SPARK-26595) Allow delegation token renewal without a keytab

2019-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26595: -- Assignee: Marcelo Vanzin > Allow delegation token renewal without a key

[jira] [Resolved] (SPARK-26595) Allow delegation token renewal without a keytab

2019-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26595. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23525 [https

[spark] branch master updated: [SPARK-26595][CORE] Allow credential renewal based on kerberos ticket cache.

2019-01-28 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 2a67dbf [SPARK-26595][CORE] Allow credential

Re: Livy-0.6 release?

2019-01-28 Thread Marcelo Vanzin
There are a couple of patches under review that are currently blocking the release. Once those are done, we can work on releasing 0.6. On Mon, Jan 28, 2019 at 11:18 AM Roger Liu wrote: > > Hey there, > > > > I’m wondering if we have a timeline for releasing Livy-0.6? Its been a year > since the

Re: JAPICMP Collections-4.3-RC2 report pdf (Was: Re: [VOTE][RC2] Commons collections 4.3)

2019-01-28 Thread Marcelo Vanzin
On Mon, Jan 28, 2019 at 11:36 AM sebb wrote: > > Haven't looked at the code, but if it's being compiled for java 8, and > > the new methods have a default implementation, then it's fine. clirr > > just complains because it's too old to know about default methods. > > I don't think so. > > I think

Re: JAPICMP Collections-4.3-RC2 report pdf (Was: Re: [VOTE][RC2] Commons collections 4.3)

2019-01-28 Thread Marcelo Vanzin
arcelo - Many thanks...Yes. That makes sense. Thanks. Seems like this > release should be a -1 then because we’re breaking BC without a major version > change. Right?? > > -Rob > > > > On Jan 28, 2019, at 2:07 PM, Marcelo Vanzin > > wrote: > > > > On Mon,

Re: JAPICMP Collections-4.3-RC2 report pdf (Was: Re: [VOTE][RC2] Commons collections 4.3)

2019-01-28 Thread Marcelo Vanzin
On Mon, Jan 28, 2019 at 11:01 AM Rob Tompkins wrote: > Before I vote on the the thread, does adding a method to an interface cause > BC to break? I would think not. All of the clirr errors are merely additions. > Further the JAPICMP report confirms this. Existing classes that implement the inte

[jira] [Resolved] (SPARK-26432) Not able to connect Hbase 2.1 service Getting NoSuchMethodException while trying to obtain token from Hbase 2.1 service.

2019-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26432. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23429 [https

[jira] [Assigned] (SPARK-26432) Not able to connect Hbase 2.1 service Getting NoSuchMethodException while trying to obtain token from Hbase 2.1 service.

2019-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26432: -- Assignee: Sujith > Not able to connect Hbase 2.1 service Gett

[spark] branch master updated: [SPARK-26432][CORE] Obtain HBase delegation token operation compatible with HBase 2.x.x version API

2019-01-28 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new dfed439 [SPARK-26432][CORE] Obtain HBase

[jira] [Resolved] (SPARK-25825) Kerberos Support for Long Running Jobs in Kubernetes

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25825. Resolution: Won't Do I'm going to close this for now since there isn't

[jira] [Assigned] (SPARK-25825) Kerberos Support for Long Running Jobs in Kubernetes

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25825: -- Assignee: (was: Marcelo Vanzin) > Kerberos Support for Long Running Jobs

[jira] [Assigned] (SPARK-25825) Kerberos Support for Long Running Jobs in Kubernetes

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25825: -- Assignee: Marcelo Vanzin > Kerberos Support for Long Running Jobs in Kuberne

[jira] [Resolved] (SPARK-25751) Unit Testing for Kerberos Support for Spark on Kubernetes

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25751. Resolution: Won't Do Other changes in this area introduced unit tests already, so

[jira] [Resolved] (SPARK-23078) Allow Submitting Spark Thrift Server in Cluster Mode

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23078. Resolution: Won't Do No answer to my question, assuming use case is now covered. &g

[jira] [Created] (SPARK-26733) Clean up entrypoint.sh

2019-01-25 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26733: -- Summary: Clean up entrypoint.sh Key: SPARK-26733 URL: https://issues.apache.org/jira/browse/SPARK-26733 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-26732) Flaky test: SparkContextInfoSuite.getRDDStorageInfo only reports on RDDs that actually persist data

2019-01-25 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26732: -- Summary: Flaky test: SparkContextInfoSuite.getRDDStorageInfo only reports on RDDs that actually persist data Key: SPARK-26732 URL: https://issues.apache.org/jira/browse/SPARK

[jira] [Commented] (SPARK-23411) Deprecate SparkContext.getRDDStorageInfo

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752734#comment-16752734 ] Marcelo Vanzin commented on SPARK-23411: Hmm, removing this method is a li

[jira] [Commented] (SPARK-23411) Deprecate SparkContext.getRDDStorageInfo

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752661#comment-16752661 ] Marcelo Vanzin commented on SPARK-23411: I saw the test for this method

[jira] [Assigned] (SPARK-23385) Allow SparkUITab to be customized adding in SparkConf and loaded when creating SparkUI

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23385: -- Assignee: Marcelo Vanzin > Allow SparkUITab to be customized adding in SparkConf

[jira] [Assigned] (SPARK-22404) Provide an option to use unmanaged AM in yarn-client mode

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-22404: -- Assignee: Devaraj K > Provide an option to use unmanaged AM in yarn-client m

[jira] [Assigned] (SPARK-23385) Allow SparkUITab to be customized adding in SparkConf and loaded when creating SparkUI

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23385: -- Assignee: (was: Marcelo Vanzin) > Allow SparkUITab to be customized adding

[jira] [Resolved] (SPARK-22404) Provide an option to use unmanaged AM in yarn-client mode

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22404. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 19616 [https

[spark] branch master updated: [SPARK-22404][YARN] Provide an option to use unmanaged AM in yarn-client mode

2019-01-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f06bc0c [SPARK-22404][YARN] Provide an option

[jira] [Resolved] (SPARK-26254) Move delegation token providers into a separate project

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26254. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23499 [https

[jira] [Assigned] (SPARK-26254) Move delegation token providers into a separate project

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26254: -- Assignee: Gabor Somogyi > Move delegation token providers into a separate proj

[spark] branch master updated: [SPARK-26254][CORE] Extract Hive + Kafka dependencies from Core.

2019-01-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 773efed [SPARK-26254][CORE] Extract Hive

[jira] [Resolved] (SPARK-26694) Console progress bar not showing in 3.0

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26694. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23618 [https

[spark] branch master updated: [SPARK-26694][CORE] Progress bar should be enabled by default for spark-shell

2019-01-25 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b484490 [SPARK-26694][CORE] Progress bar should

[jira] [Assigned] (SPARK-26694) Console progress bar not showing in 3.0

2019-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26694: -- Assignee: Ankur Gupta > Console progress bar not showing in

[jira] [Assigned] (SPARK-13587) Support virtualenv in PySpark

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-13587: -- Assignee: (was: Marcelo Vanzin) > Support virtualenv in PySp

[jira] [Assigned] (SPARK-13587) Support virtualenv in PySpark

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-13587: -- Assignee: Marcelo Vanzin (was: Jeff Zhang) > Support virtualenv in PySp

[jira] [Assigned] (SPARK-26697) ShuffleBlockFetcherIterator can log block sizes in addition to num blocks

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26697: -- Assignee: Imran Rashid > ShuffleBlockFetcherIterator can log block sizes in addit

[spark] branch master updated: [SPARK-26697][CORE] Log local & remote block sizes.

2019-01-24 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 3699763 [SPARK-26697][CORE] Log local & re

[jira] [Resolved] (SPARK-26697) ShuffleBlockFetcherIterator can log block sizes in addition to num blocks

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26697. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23621 [https

[jira] [Assigned] (SPARK-26530) Validate heartheat arguments in HeartbeatReceiver

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26530: -- Assignee: liupengcheng > Validate heartheat arguments in HeartbeatRecei

[jira] [Resolved] (SPARK-26530) Validate heartheat arguments in HeartbeatReceiver

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26530. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23445 [https

[spark] branch master updated: [SPARK-26530][CORE] Validate heartheat arguments in HeartbeatReceiver

2019-01-24 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8d667c5 [SPARK-26530][CORE] Validate heartheat

[jira] [Updated] (SPARK-26682) Task attempt ID collision causes lost data

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26682: --- Fix Version/s: 2.3.3 > Task attempt ID collision causes lost d

[spark] branch branch-2.3 updated: [SPARK-26682][SQL] Use taskAttemptID instead of attemptNumber for Had…

2019-01-24 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new ded902c [SPARK-26682][SQL] Use

[jira] [Commented] (SPARK-25590) kubernetes-model-2.0.0.jar masks default Spark logging config

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751425#comment-16751425 ] Marcelo Vanzin commented on SPARK-25590: I filed https://github.com/fabri

Re: How to force-quit a Spark application?

2019-01-24 Thread Marcelo Vanzin
Hi, On Tue, Jan 22, 2019 at 11:30 AM Pola Yao wrote: > "Thread-1" #19 prio=5 os_prio=0 tid=0x7f9b6828e800 nid=0x77cb waiting on > condition [0x7f9a123e3000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0

[jira] [Resolved] (SPARK-26687) Building Spark Images has non-intuitive behaviour with paths to custom Dockerfiles

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26687. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23613 [https

[jira] [Assigned] (SPARK-26687) Building Spark Images has non-intuitive behaviour with paths to custom Dockerfiles

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26687: -- Assignee: Rob Vesse > Building Spark Images has non-intuitive behaviour with pa

[spark] branch master updated: [SPARK-26687][K8S] Fix handling of custom Dockerfile paths

2019-01-24 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 69dab94 [SPARK-26687][K8S] Fix handling of

[jira] [Resolved] (SPARK-26717) Support PodPriority for spark driver and executor on kubernetes

2019-01-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26717. Resolution: Not A Problem Pretty sure this is covered by pod templates (SPARK-24434

Re: [VOTE] Release Apache Spark 2.3.3 (RC1)

2019-01-23 Thread Marcelo Vanzin
-1 too. I just upgraded https://issues.apache.org/jira/browse/SPARK-26682 to blocker. It's a small fix and we should make it in 2.3.3. On Thu, Jan 17, 2019 at 6:49 PM Takeshi Yamamuro wrote: > > Please vote on releasing the following candidate as Apache Spark version > 2.3.3. > > The vote is op

[jira] [Updated] (SPARK-26682) Task attempt ID collision causes lost data

2019-01-23 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26682: --- Target Version/s: 2.3.3, 2.4.1 Labels: data-loss (was: ) Priority

[jira] [Commented] (SPARK-26704) docker-image-tool.sh should copy custom Dockerfiles into the build context for inclusion in images

2019-01-23 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750337#comment-16750337 ] Marcelo Vanzin commented on SPARK-26704: bq. they will be present and

[jira] [Commented] (SPARK-26704) docker-image-tool.sh should copy custom Dockerfiles into the build context for inclusion in images

2019-01-23 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750249#comment-16750249 ] Marcelo Vanzin commented on SPARK-26704: You mentioned in the discussion

[jira] [Assigned] (SPARK-26605) New executors failing with expired tokens in client mode

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26605: -- Assignee: Marcelo Vanzin > New executors failing with expired tokens in client m

[jira] [Resolved] (SPARK-26605) New executors failing with expired tokens in client mode

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26605. Resolution: Fixed Fix Version/s: 2.4.1 Issue resolved by pull request 23523 [https

[spark] branch branch-2.4 updated: [SPARK-26605][YARN] Update AM's credentials when creating tokens.

2019-01-22 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 10d7713 [SPARK-26605][YARN] Update AM&#

[jira] [Created] (SPARK-26694) Console progress bar not showing in 3.0

2019-01-22 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26694: -- Summary: Console progress bar not showing in 3.0 Key: SPARK-26694 URL: https://issues.apache.org/jira/browse/SPARK-26694 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-26685) Building Spark Images with latest Docker does not honour spark_uid build argument

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26685. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23611 [https

[jira] [Assigned] (SPARK-26685) Building Spark Images with latest Docker does not honour spark_uid build argument

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26685: -- Assignee: Rob Vesse > Building Spark Images with latest Docker does not hon

[spark] branch master updated: [SPARK-26685][K8S] Correct placement of ARG declaration

2019-01-22 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new dc2da72 [SPARK-26685][K8S] Correct placement of

[jira] [Resolved] (SPARK-25887) Allow specifying Kubernetes context to use

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25887. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22904 [https

[jira] [Assigned] (SPARK-25887) Allow specifying Kubernetes context to use

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25887: -- Assignee: Rob Vesse > Allow specifying Kubernetes context to

[spark] branch master updated: [SPARK-25887][K8S] Configurable K8S context support

2019-01-22 Thread vanzin
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c542c24 [SPARK-25887][K8S] Configurable K8S

[jira] [Commented] (SPARK-25960) Support subpath mounting with Kubernetes

2019-01-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745353#comment-16745353 ] Marcelo Vanzin commented on SPARK-25960: I general we don't bac

Re: How to force-quit a Spark application?

2019-01-16 Thread Marcelo Vanzin
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > "

Re: How to force-quit a Spark application?

2019-01-16 Thread Marcelo Vanzin
ed many ways to > exit the spark (e.g., System.exit()), but failed. Is there an explicit way to > shutdown all the alive threads in the spark application and then quit > afterwards? > > > On Tue, Jan 15, 2019 at 2:38 PM Marcelo Vanzin wrote: >> >> You should check th

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

2019-01-15 Thread Marcelo Vanzin
+1 to that. HIVE-16391 by itself means we're giving up things like Hadoop 3, and we're also putting the burden on the Hive folks to fix a problem that we created. The current PR is basically a Spark-side fix for that bug. It does mean also upgrading Hive (which gives us Hadoop 3, yay!), but I thin

<    5   6   7   8   9   10   11   12   13   14   >