[jira] [Assigned] (LIVY-637) get NullPointerException when create database using thriftserver

2019-08-21 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/LIVY-637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-637: Assignee: mingchao zhao > get NullPointerException when create database using thriftser

[jira] [Closed] (LIVY-591) ACLs enforcement should occur on both session owner and proxy user

2019-08-21 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/LIVY-591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed LIVY-591. Resolution: Duplicate > ACLs enforcement should occur on both session owner and proxy u

[jira] [Resolved] (LIVY-592) Proxy user cannot view its session log

2019-08-21 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/LIVY-592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-592. -- Fix Version/s: 0.7.0 Assignee: Yiheng Wang Resolution: Fixed > Proxy user cannot v

[jira] [Commented] (LIVY-592) Proxy user cannot view its session log

2019-08-21 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/LIVY-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912875#comment-16912875 ] Saisai Shao commented on LIVY-592: -- Issue resolved by pull request 202 https://github

Re: Are we going to use Apache JIRA instead of Github issues

2019-08-18 Thread Saisai Shao
rch (in my opinion). I'm just one vote, though, so if most people >> prefer to move to JIRA I'm open to it. >> >> What do you think is missing compared to JIRA? >> >> On Fri, Aug 16, 2019 at 3:09 AM Saisai Shao >> wrote: >> >>> Hi Team, &

Re: Release Spark 2.3.4

2019-08-18 Thread Saisai Shao
+1 Wenchen Fan 于2019年8月19日周一 上午10:28写道: > +1 > > On Sat, Aug 17, 2019 at 3:37 PM Hyukjin Kwon wrote: > >> +1 too >> >> 2019년 8월 17일 (토) 오후 3:06, Dilip Biswal 님이 작성: >> >>> +1 >>> >>> Regards, >>> Dilip Biswal >>> Tel: 408-463-4980 >>> dbis...@us.ibm.com >>> >>> >>> >>> - Original message --

Are we going to use Apache JIRA instead of Github issues

2019-08-16 Thread Saisai Shao
Hi Team, Seems Iceberg project uses Github issues instead of JIRA. IMHO JIRA is more powerful and easy to manage, most of the Apache projects use JIRA to track everything, any plan to move to JIRA or we stick on using Github issues? Thanks Saisai

[jira] [Assigned] (LIVY-623) Implement GetTables metadata operation

2019-08-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-623: Assignee: Yiheng Wang > Implement GetTables metadata operat

[jira] [Assigned] (LIVY-625) Implement GetFunctions metadata operation

2019-08-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-625: Assignee: Yiheng Wang > Implement GetFunctions metadata operat

[jira] [Assigned] (LIVY-624) Implement GetColumns metadata operation

2019-08-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-624: Assignee: Yiheng Wang > Implement GetColumns metadata operat

[jira] [Updated] (LIVY-575) Implement missing metadata operations

2019-08-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-575: - Priority: Major (was: Minor) > Implement missing metadata operati

[jira] [Assigned] (LIVY-622) Implement GetSchemas metadata operation

2019-08-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-622: Assignee: Yiheng Wang > Implement GetSchemas metadata operat

[jira] [Resolved] (LIVY-625) Implement GetFunctions metadata operation

2019-08-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-625. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by pull request 194 [https://github.com

[jira] [Resolved] (LIVY-624) Implement GetColumns metadata operation

2019-08-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-624. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by pull request 194 [https://github.com

[jira] [Resolved] (LIVY-623) Implement GetTables metadata operation

2019-08-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-623. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by pull request 194 [https://github.com

[jira] [Resolved] (LIVY-622) Implement GetSchemas metadata operation

2019-08-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-622. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by pull request 194 [https://github.com

[jira] [Updated] (LIVY-635) Travis failed to build

2019-08-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-635: - Component/s: Build > Travis failed to build > -- > > K

[jira] [Assigned] (LIVY-635) Travis failed to build

2019-08-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-635: Assignee: jiewang > Travis failed to build > -- > >

[jira] [Resolved] (LIVY-635) Travis failed to build

2019-08-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-635. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by pull request 198 [https://github.com

[jira] [Assigned] (LIVY-620) Spark batch session always ends with success when configuration is master yarn and deploy-mode client

2019-08-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-620: Assignee: Gustavo Martin > Spark batch session always ends with success when configuration

[jira] [Resolved] (LIVY-620) Spark batch session always ends with success when configuration is master yarn and deploy-mode client

2019-08-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-620. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by pull request 192 [https://github.com

[jira] [Commented] (LIVY-616) Livy Server discovery

2019-08-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906098#comment-16906098 ] Saisai Shao commented on LIVY-616: -- I will spend time on the PR itself, but I'm

[jira] [Commented] (LIVY-616) Livy Server discovery

2019-08-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905734#comment-16905734 ] Saisai Shao commented on LIVY-616: -- I don't have the objections about it, just

[jira] [Commented] (LIVY-616) Livy Server discovery

2019-08-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904966#comment-16904966 ] Saisai Shao commented on LIVY-616: -- I think what I mean is to use http endpoint, si

[jira] [Commented] (LIVY-616) Livy Server discovery

2019-08-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904922#comment-16904922 ] Saisai Shao commented on LIVY-616: -- Yes, I was asking how to use HTTP request dire

[jira] [Commented] (LIVY-616) Livy Server discovery

2019-08-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904908#comment-16904908 ] Saisai Shao commented on LIVY-616: -- I think I was asking if user directly using REST

[jira] [Commented] (LIVY-616) Livy Server discovery

2019-08-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904863#comment-16904863 ] Saisai Shao commented on LIVY-616: -- Two questions about your design: 1. How do

Re: Creating session in livy with jars parameter

2019-08-11 Thread Saisai Shao
va for example) - session is > in *IDLE* status > >> {"kind":"spark", "jars":["local:/path_to_guava_jar/guava.jar"]} > > > I don't understand what can be the difference between my jar and some > other jars... > > On Sun, Aug 11

Re: Creating session in livy with jars parameter

2019-08-11 Thread Saisai Shao
Would you mind listing the steps to reproduce your issue, and how do you use REST APIs? Thanks Saisai Pavel Sivak 于2019年8月10日周六 上午11:01写道: > Hi, > My idea is to create livy session with my library in class path using > "jars" parameter. > I'm using REST API to create a session. After sending PO

Re: Pre-registering UDTs / UDFs in Livy session

2019-08-11 Thread Saisai Shao
Unfortunately there's no such mechanism to inject custom code when session is started in Livy side. I think you can add some code in Spark side, Spark has a listener hook "SparkListener", in which there has a hook ` onApplicationStart`, this hook will be called immediately after application is star

[jira] [Assigned] (LIVY-547) Livy kills session after livy.server.session.timeout even if the session is active

2019-08-08 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-547: Assignee: shanyu zhao > Livy kills session after livy.server.session.timeout even if the session

[jira] [Resolved] (LIVY-547) Livy kills session after livy.server.session.timeout even if the session is active

2019-08-08 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-547. -- Resolution: Fixed Fix Version/s: 0.7.0 > Livy kills session after livy.server.session.time

[jira] [Commented] (LIVY-547) Livy kills session after livy.server.session.timeout even if the session is active

2019-08-08 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903616#comment-16903616 ] Saisai Shao commented on LIVY-547: -- Issue resolved by pull request 190 https://github

Re: Any plan to support update, delete and others

2019-08-08 Thread Saisai Shao
e, Spark supports the latest features available in DataSourceV2, > and will continue to. In fact, we're adding features to DSv2 based on what > we've built internally at Netflix to support Iceberg. > > On Wed, Aug 7, 2019 at 7:03 PM Saisai Shao wrote: > >> Thanks a lo

Re: Two newbie question about Iceberg

2019-08-08 Thread Saisai Shao
gt; missing automatically? > > On Wed, Aug 7, 2019 at 7:13 PM Saisai Shao wrote: > >> Thanks guys for your reply. >> >> I didn't do anything special, I don't even have a configured Hive. I just >> simply put the iceberg (assembly) jar into Spark and start a local

Re: Iceberg in Spark 3.0.0

2019-08-07 Thread Saisai Shao
IMHO I agree that we should have a branch to track the changes for Spark 3.0.0. Spark 3.0.0 has several changes regarding to DataSource V2, it would be better to evaluate the changes and do the design by also considering 3.0 changes. My two cents :) Best regards, Saisai Edgar Rodriguez 于2019年8月

Re: Two newbie question about Iceberg

2019-08-07 Thread Saisai Shao
der if a newer version of Hive would avoid this problem? What version > are you linking with? > > > > On Tue, Aug 6, 2019 at 8:59 PM Saisai Shao > wrote: > > Hi team, > > > > I just met some issues when trying Iceberg with quick start guide. Not > sure if it is pr

Re: Any plan to support update, delete and others

2019-08-07 Thread Saisai Shao
y to call this yet. Same for > MERGE > INTO, open source Spark doesn’t support the operation yet. We’re also > working on building support into Spark as we go. > > I hope that helps! > > On Wed, Aug 7, 2019 at 4:25 AM Saisai Shao wrote: > >> Hi team, >> >&

Any plan to support update, delete and others

2019-08-07 Thread Saisai Shao
Hi team, Delta lake project recently announced version 0.3.0, which added several new features in API level, like update, delete, merge, vacuum, etc. May I ask is there any plan to add such features in Iceberg? Thanks Saisai

Re: Podling Report Reminder - August 2019

2019-08-07 Thread Saisai Shao
Oh, sorry, let me check it. Also it would grateful if any mentor could sign-off the report. Thanks Saisai Justin Mclean 于2019年8月7日周三 下午3:39写道: > Hi, > > Thanks for submitting your report but I notice you didn't answer the "Have > your mentors been helpful and responsive?" question. It would be

Two newbie question about Iceberg

2019-08-06 Thread Saisai Shao
Hi team, I just met some issues when trying Iceberg with quick start guide. Not sure if it is proper to send this to @dev mail list (seems there's no user mail list). One issue is that seems current Iceberg cannot run with embedded metastore. It will throw an exception. Is this an on-purpose beha

[jira] [Commented] (LIVY-616) Livy Server discovery

2019-08-06 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900749#comment-16900749 ] Saisai Shao commented on LIVY-616: -- I see your point. Let me check the design and

[jira] [Commented] (LIVY-616) Livy Server discovery

2019-08-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900602#comment-16900602 ] Saisai Shao commented on LIVY-616: -- [~oshevchenko] you can either push an empty commi

[jira] [Assigned] (SPARK-28475) Add regex MetricFilter to GraphiteSink

2019-08-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-28475: --- Assignee: Nick Karpov > Add regex MetricFilter to GraphiteS

[jira] [Updated] (SPARK-28475) Add regex MetricFilter to GraphiteSink

2019-08-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-28475: Priority: Minor (was: Major) > Add regex MetricFilter to GraphiteS

[jira] [Resolved] (SPARK-28475) Add regex MetricFilter to GraphiteSink

2019-08-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-28475. - Resolution: Resolved Fix Version/s: 3.0.0 > Add regex MetricFilter to GraphiteS

[jira] [Commented] (SPARK-28475) Add regex MetricFilter to GraphiteSink

2019-08-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898749#comment-16898749 ] Saisai Shao commented on SPARK-28475: - This is resolved via https://github

[jira] [Closed] (LIVY-597) Upgrade Livy guava dependency

2019-07-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed LIVY-597. Resolution: Duplicate > Upgrade Livy guava depende

[jira] [Assigned] (LIVY-587) Remove Guava dependency

2019-07-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-587: Assignee: Saisai Shao (was: jiewang) > Remove Guava depende

[jira] [Assigned] (LIVY-587) Remove Guava dependency

2019-07-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-587: Assignee: jiewang (was: Saisai Shao) > Remove Guava depende

[jira] [Assigned] (LIVY-587) Remove Guava dependency

2019-07-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-587: Assignee: jiewang > Remove Guava dependency > --- > >

[jira] [Resolved] (LIVY-587) Remove Guava dependency

2019-07-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-587. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by pull request 181 [https://github.com

[jira] [Assigned] (LIVY-613) Livy can't handle the java.sql.Date type correctly

2019-07-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-613: Assignee: wyp > Livy can't handle the java.sql.Date type c

[jira] [Resolved] (LIVY-613) Livy can't handle the java.sql.Date type correctly

2019-07-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-613. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by pull request 186 [https://github.com

[jira] [Commented] (LIVY-575) Implement missing metadata operations

2019-07-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890656#comment-16890656 ] Saisai Shao commented on LIVY-575: -- Yes, seems like resolving wrong jiras. > Im

[jira] [Updated] (LIVY-575) Implement missing metadata operations

2019-07-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-575: - Fix Version/s: (was: 0.7.0) > Implement missing metadata operati

[jira] [Reopened] (LIVY-575) Implement missing metadata operations

2019-07-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reopened LIVY-575: -- > Implement missing metadata operations > - > >

[jira] [Updated] (LIVY-609) LDAP auth for Livy thriftserver

2019-07-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-609: - Priority: Major (was: Minor) > LDAP auth for Livy thriftser

[jira] [Updated] (LIVY-609) LDAP auth for Livy thriftserver

2019-07-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-609: - Fix Version/s: (was: 0.7.0) > LDAP auth for Livy thriftser

[jira] [Assigned] (LIVY-610) optimization for windows environment build.

2019-07-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-610: Assignee: dockerzhang > optimization for windows environment bu

[jira] [Resolved] (LIVY-610) optimization for windows environment build.

2019-07-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-610. -- Resolution: Fixed Issue resolved by pull request 184 [https://github.com/apache/incubator-livy/pull/184

[jira] [Commented] (LIVY-575) Implement missing metadata operations

2019-07-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887773#comment-16887773 ] Saisai Shao commented on LIVY-575: -- Hi [~mgaido], we have 3 colleagues planning to

[jira] [Assigned] (SPARK-28106) Spark SQL add jar with wrong hdfs path, SparkContext still add it to jar path ,and cause Task Failed

2019-07-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-28106: --- Assignee: angerszhu > Spark SQL add jar with wrong hdfs path, SparkContext still add it

[jira] [Resolved] (SPARK-28106) Spark SQL add jar with wrong hdfs path, SparkContext still add it to jar path ,and cause Task Failed

2019-07-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-28106. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24909 [https

[jira] [Updated] (SPARK-28106) Spark SQL add jar with wrong hdfs path, SparkContext still add it to jar path ,and cause Task Failed

2019-07-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-28106: Component/s: Spark Core > Spark SQL add jar with wrong hdfs path, SparkContext still add it

[jira] [Resolved] (LIVY-582) python test_create_new_session_without_default_config test fails consistently

2019-07-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-582. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by pull request 180 [https://github.com

[jira] [Assigned] (LIVY-582) python test_create_new_session_without_default_config test fails consistently

2019-07-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-582: Assignee: Yiheng Wang > python test_create_new_session_without_default_config test fa

[jira] [Assigned] (LIVY-603) upgrade build spark version to 2.4.3

2019-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-603: Assignee: Yiheng Wang > upgrade build spark version to 2.

[jira] [Resolved] (LIVY-603) upgrade build spark version to 2.4.3

2019-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-603. -- Resolution: Fixed Issue resolved by pull request 179 [https://github.com/apache/incubator-livy/pull/179

[jira] [Commented] (LIVY-587) Remove Guava dependency

2019-07-10 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882007#comment-16882007 ] Saisai Shao commented on LIVY-587: -- I would suggest to remove guava rather than upg

[jira] [Comment Edited] (LIVY-587) Remove Guava dependency

2019-07-10 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882007#comment-16882007 ] Saisai Shao edited comment on LIVY-587 at 7/10/19 12:40 PM: ---

[jira] [Assigned] (SPARK-28202) [Core] [Test] Avoid noises of system props in SparkConfSuite

2019-07-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-28202: --- Assignee: ShuMing Li > [Core] [Test] Avoid noises of system props in SparkConfSu

[jira] [Resolved] (SPARK-28202) [Core] [Test] Avoid noises of system props in SparkConfSuite

2019-07-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-28202. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24998 [https

[jira] [Commented] (SPARK-25299) Use remote storage for persisting shuffle data

2019-07-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16876070#comment-16876070 ] Saisai Shao commented on SPARK-25299: - Better to post a pdf version [~mc

[jira] [Commented] (SPARK-25299) Use remote storage for persisting shuffle data

2019-06-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874703#comment-16874703 ] Saisai Shao commented on SPARK-25299: - Votes were passed, so what is our plan

Re: [VOTE][SPARK-25299] SPIP: Shuffle Storage API

2019-06-16 Thread Saisai Shao
+1 (binding) Thanks Saisai Imran Rashid 于2019年6月15日周六 上午3:46写道: > +1 (binding) > > I think this is a really important feature for spark. > > First, there is already a lot of interest in alternative shuffle storage > in the community. There is already a lot of interest in alternative > shuffle

[jira] [Updated] (SPARK-25299) Use remote storage for persisting shuffle data

2019-06-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-25299: Labels: SPIP (was: ) > Use remote storage for persisting shuffle d

Re: [DISCUSS][SPARK-25299] SPIP: Shuffle storage API

2019-06-12 Thread Saisai Shao
I think maybe we could start a vote on this SPIP. This has been discussed for a while, and the current doc is pretty complete as for now. Also we saw lots of demands in the community about building their own shuffle storage. Thanks Saisai Imran Rashid 于2019年6月11日周二 上午3:27写道: > I would be happy

[jira] [Created] (SPARK-27996) Spark UI redirect will be failed behind the https reverse proxy

2019-06-10 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-27996: --- Summary: Spark UI redirect will be failed behind the https reverse proxy Key: SPARK-27996 URL: https://issues.apache.org/jira/browse/SPARK-27996 Project: Spark

Re: [DISCUSS][SPARK-25299] SPIP: Shuffle storage API

2019-06-10 Thread Saisai Shao
I'm currently working with MemVerge on the Splash project (one implementation of remote shuffle storage) and followed this ticket for a while. I would like to be a shepherd if no one else volunteered to be. Best regards, Saisai Matt Cheah 于2019年6月6日周四 上午8:33写道: > Hi everyone, > > > > I wanted t

Re: Support for Livy with Scala 2.12

2019-06-05 Thread Saisai Shao
migration that was done an year ago and it > looks like it is mostly Pom changes. If that’s not correct than I might > need help to perform the upgrade. > > Do you have bandwidth to make this change? > > > > > From: Saisai Shao mailto:sai.sai.s...@gmail.com&g

Re: Support for Livy with Scala 2.12

2019-06-04 Thread Saisai Shao
ate 2.12 module? Is that just a > change in POM files or code change is required? > > We have release planned for July to upgrade Jupyter and Livy to utilize > spark 2.4.2. This is blocking us from upgrade. > From: Saisai Shao mailto:sai.sai.s...@gmail.com>> > Date: Monday,

Re: Support for Livy with Scala 2.12

2019-06-03 Thread Saisai Shao
Like what we did before to support both Scala 2.10 and 2.11 in Livy, I think we should also have a new module to support 2.12. 于2019年6月4日周二 上午7:40写道: > Yes, the spark binary we downloaded is built with default Scala 2.12. We > want to use databricks delta which I think only support Scala 2.12.

[jira] [Commented] (SPARK-15348) Hive ACID

2019-05-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844750#comment-16844750 ] Saisai Shao commented on SPARK-15348: - No, it doesn't support hive ACID, i

[jira] [Commented] (SPARK-15348) Hive ACID

2019-05-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844597#comment-16844597 ] Saisai Shao commented on SPARK-15348: - I think delta lake project is exactly

[jira] [Comment Edited] (SPARK-15348) Hive ACID

2019-05-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844597#comment-16844597 ] Saisai Shao edited comment on SPARK-15348 at 5/21/19 7:3

Re: Dynamic metric names

2019-05-06 Thread Saisai Shao
s by means of alternative software (e.g. > grafana) - currently it's hardly possible to know the actual number of > jobs, stages, tasks and their names and IDs in advance to register all the > corresponding metrics statically. > > > Kind Regards, > Sergey > > >

Re: Dynamic metric names

2019-05-06 Thread Saisai Shao
I remembered there was a PR about doing similar thing ( https://github.com/apache/spark/pull/18406). From my understanding, this seems like a quite specific requirement, it may requires code change to support your needs. Thanks Saisai Sergey Zhemzhitsky 于2019年5月4日周六 下午4:44写道: > Hello Spark User

Re: Livy-0.6 release?

2019-03-12 Thread Saisai Shao
I can also help to release a new version. My only concern is that how mature the thrift module is, shall we enable it by default or leave it disabled? Thanks Saisai Jeff Zhang 于2019年3月12日周二 上午10:54写道: > Thanks Marcelo, I can help to test it in zeppelin side which use livy as > one interpreter.

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

2019-03-06 Thread Saisai Shao
Do we have other block/critical issues for Spark 2.4.1 or waiting something to be fixed? I roughly searched the JIRA, seems there's no block/critical issues marked for 2.4.1. Thanks Saisai shane knapp 于2019年3月7日周四 上午4:57写道: > i'll be popping in to the sig-big-data meeting on the 20th to talk ab

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

2019-03-05 Thread Saisai Shao
Hi DB, I saw that we already have 6 RCs, but the vote I can search by now was RC2, were they all canceled? Thanks Saisai DB Tsai 于2019年2月22日周五 上午4:51写道: > I am cutting a new rc4 with fix from Felix. Thanks. > > Sincerely, > > DB Tsai > --

Re: Making travis logs less verbose

2019-02-11 Thread Saisai Shao
ason why two mvn builds are >needed? > > install: - mvn $MVN_FLAG install -Dskip -DskipTests -DskipITs > -Dmaven.javadoc.skip=true -B -V > script: - mvn $MVN_FLAG verify -e > > > On Mon, Feb 11, 2019 at 5:52 PM Saisai Shao > wrote: > > > Problem is that th

Re: Making travis logs less verbose

2019-02-11 Thread Saisai Shao
Problem is that there's no better way to get detailed log on travis without printing in out on screen. I was struggling on it when debugging on travis. Thanks Saisai Meisam Fathi 于2019年2月9日周六 下午12:19写道: > This may do the trick for maven > > mvn -Dorg.slf4j.simpleLogger.defaultLogLevel=warn ...

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2019-01-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750872#comment-16750872 ] Saisai Shao commented on SPARK-24615: - I'm really sorry about the dela

[jira] [Commented] (SPARK-26512) Spark 2.4.0 is not working with Hadoop 2.8.3 in windows 10

2019-01-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16734072#comment-16734072 ] Saisai Shao commented on SPARK-26512: - This seems like a Netty version pro

Re: [VOTE] Move source repo to gitbox

2019-01-03 Thread Saisai Shao
+1 Luciano Resende 于2019年1月4日周五 上午8:26写道: > +1 > > On Thu, Jan 3, 2019 at 15:57 Marcelo Vanzin > wrote: > > > Creating a formal vote for this. I don't think we have a choice but > > they seem to request a vote anyway. This will be lazy consensus, I > > plan to create an infra ticket for the mig

[jira] [Commented] (SPARK-26512) Spark 2.4.0 is not working with Hadoop 2.8.3 in windows 10?

2019-01-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16733740#comment-16733740 ] Saisai Shao commented on SPARK-26512: - Please list the problems you saw, any lo

[jira] [Updated] (SPARK-26457) Show hadoop configurations in HistoryServer environment tab

2019-01-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-26457: Priority: Minor (was: Major) > Show hadoop configurations in HistoryServer environment

[jira] [Resolved] (SPARK-26516) zeppelin with spark on mesos: environment variable setting

2019-01-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-26516. - Resolution: Invalid > zeppelin with spark on mesos: environment variable sett

[jira] [Updated] (SPARK-26513) Trigger GC on executor node idle

2019-01-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-26513: Fix Version/s: (was: 3.0.0) > Trigger GC on executor node i

<    1   2   3   4   5   6   7   8   9   10   >