[jira] [Commented] (SPARK-23982) NoSuchMethodException: There is no startCredentialUpdater method in the object YarnSparkHadoopUtil

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441881#comment-16441881 ] Saisai Shao commented on SPARK-23982: - This method should be existed. Would

[jira] [Created] (LIVY-458) Upgrade jackson version from 2.9.2 to 2.9.5

2018-04-17 Thread Saisai Shao (JIRA)
Saisai Shao created LIVY-458: Summary: Upgrade jackson version from 2.9.2 to 2.9.5 Key: LIVY-458 URL: https://issues.apache.org/jira/browse/LIVY-458 Project: Livy Issue Type: Bug

[jira] [Resolved] (LIVY-457) PySpark `sqlContext.sparkSession` incorrect on Spark 2.x

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-457. -- Resolution: Fixed Fix Version/s: 0.6.0 0.5.1 Issue resolved by pull request 86

[jira] [Assigned] (LIVY-457) PySpark `sqlContext.sparkSession` incorrect on Spark 2.x

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-457: Assignee: Saisai Shao > PySpark `sqlContext.sparkSession` incorrect on Spark

[jira] [Commented] (LIVY-457) PySpark `sqlContext.sparkSession` incorrect on Spark 2.x

2018-04-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440460#comment-16440460 ] Saisai Shao commented on LIVY-457: -- I made a PR about this issue (https://github

[jira] [Commented] (LIVY-456) IllegalStateException when interactive session is still starting

2018-04-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440444#comment-16440444 ] Saisai Shao commented on LIVY-456: -- Why don't you make a PR about

[jira] [Commented] (LIVY-457) PySpark `sqlContext.sparkSession` incorrect on Spark 2.x

2018-04-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440435#comment-16440435 ] Saisai Shao commented on LIVY-457: -- Yes, we also met this issue today. The signatur

[jira] [Resolved] (SPARK-23956) Use effective RPC port in AM registration

2018-04-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23956. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21047 [https

[jira] [Assigned] (SPARK-23956) Use effective RPC port in AM registration

2018-04-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23956: --- Assignee: Gera Shegalov > Use effective RPC port in AM registrat

Re: How to submit some code segment to existing SparkContext

2018-04-11 Thread Saisai Shao
Maybe you can try Livy (http://livy.incubator.apache.org/). Thanks Jerry 2018-04-11 15:46 GMT+08:00 杜斌 : > Hi, > > Is there any way to submit some code segment to the existing SparkContext? > Just like a web backend, send some user code to the Spark to run, but the > initial SparkContext takes t

[jira] [Resolved] (LIVY-455) Livy Spark SQL "not a primitive class java.math.BigDecimal" when querying decimal columns

2018-04-10 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-455. -- Resolution: Fixed Fix Version/s: 0.6.0 0.5.1 Issue resolved by pull request 85

Re: spark application running in yarn client mode is slower than in local mode.

2018-04-09 Thread Saisai Shao
> > In yarn mode, only two executor are assigned to process the task, since > one executor can process one task only, they need 6 min in total. > This is not true. You should set --executor-cores/--num-executors to increase the task parallelism for executor. To be fair, Spark application should ha

[jira] [Assigned] (LIVY-455) Livy Spark SQL "not a primitive class java.math.BigDecimal" when querying decimal columns

2018-04-08 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-455: Assignee: Saisai Shao > Livy Spark SQL "not a primitive class java.math.BigDecimal"

[jira] [Commented] (LIVY-455) Livy Spark SQL "not a primitive class java.math.BigDecimal" when querying decimal columns

2018-04-08 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429944#comment-16429944 ] Saisai Shao commented on LIVY-455: -- Thanks for the reporting, seems like there's

Re: Hadoop 3 support

2018-04-02 Thread Saisai Shao
Yes, the main blocking issue is the hive version used in Spark (1.2.1.spark) doesn't support run on Hadoop 3. Hive will check the Hadoop version in the runtime [1]. Besides this I think some pom changes should be enough to support Hadoop 3. If we want to use Hadoop 3 shaded client jar, then the po

[jira] [Created] (SPARK-23845) Continuous rate source uses different offset format

2018-04-02 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-23845: --- Summary: Continuous rate source uses different offset format Key: SPARK-23845 URL: https://issues.apache.org/jira/browse/SPARK-23845 Project: Spark Issue Type

Re: 回复: Welcome Zhenhua Wang as a Spark committer

2018-04-02 Thread Saisai Shao
Congrats, Zhenhua! 2018-04-02 16:57 GMT+08:00 Takeshi Yamamuro : > Congrats, Zhenhua! > > On Mon, Apr 2, 2018 at 4:13 PM, Ted Yu wrote: > >> Congratulations, Zhenhua >> >> Original message >> From: 雨中漫步 <601450...@qq.com> >> Date: 4/1/18 11:30 PM (GMT-08:00) >> To: Yuanjian Li

[jira] [Created] (SPARK-23844) Socket Stream recovering from checkpoint will throw exception

2018-04-02 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-23844: --- Summary: Socket Stream recovering from checkpoint will throw exception Key: SPARK-23844 URL: https://issues.apache.org/jira/browse/SPARK-23844 Project: Spark

[jira] [Resolved] (LIVY-454) Creating batch sessions seems to be synchronous. Is there any plan to make it asynchronous ?

2018-04-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-454. -- Resolution: Won't Fix > Creating batch sessions seems to be synchronous. Is there any plan to

[jira] [Commented] (LIVY-454) Creating batch sessions seems to be synchronous. Is there any plan to make it asynchronous ?

2018-04-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16421902#comment-16421902 ] Saisai Shao commented on LIVY-454: -- No, I don't think your problem can be sol

[jira] [Resolved] (SPARK-23767) DirectStream is producing the incorrect type of message

2018-03-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23767. - Resolution: Not A Problem > DirectStream is producing the incorrect type of mess

[jira] [Commented] (SPARK-23767) DirectStream is producing the incorrect type of message

2018-03-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16420291#comment-16420291 ] Saisai Shao commented on SPARK-23767: - This seems not a Spark issue, more li

[jira] [Assigned] (SPARK-23743) IsolatedClientLoader.isSharedClass returns an unindented result against `slf4j` keyword

2018-03-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23743: --- Assignee: Jongyoul Lee > IsolatedClientLoader.isSharedClass returns an unindented res

[jira] [Resolved] (SPARK-23743) IsolatedClientLoader.isSharedClass returns an unindented result against `slf4j` keyword

2018-03-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23743. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20860 [https

Re: Podling Report Reminder - April 2018

2018-03-28 Thread Saisai Shao
Thanks a lot Alex, LGTM. Best regards, Jerry 2018-03-29 4:48 GMT+08:00 Alex Bozarth : > If a mentor wants to take a look I've updated the report, as I mentioned > before it's really short this time around. > > > *Alex Bozarth* > Software Engineer > Spark Technology Center > -

[jira] [Commented] (SPARK-23807) Add Hadoop 3 profile with relevant POM fix ups, cloud-storage artifacts and binding

2018-03-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418298#comment-16418298 ] Saisai Shao commented on SPARK-23807: - Hi [~ste...@apache.org], this is dup of S

[jira] [Commented] (SPARK-22876) spark.yarn.am.attemptFailuresValidityInterval does not work correctly

2018-03-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413412#comment-16413412 ] Saisai Shao commented on SPARK-22876: - Yes, that's the key problem. Cur

[jira] [Resolved] (SPARK-23787) SparkSubmitSuite::"download remote resource if it is not supported by yarn" fails on Hadoop 2.9

2018-03-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23787. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20895 [https

[jira] [Assigned] (SPARK-23787) SparkSubmitSuite::"download remote resource if it is not supported by yarn" fails on Hadoop 2.9

2018-03-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23787: --- Assignee: Marcelo Vanzin > SparkSubmitSuite::"download remote resource if i

Re: Spark and Accumulo Delegation tokens

2018-03-23 Thread Saisai Shao
er) > > // Filter out providers for which > spark.security.credentials.{service}.enabled is false. > providers > .filter { p => isServiceEnabled(p.serviceName) } > .map { p => (p.serviceName, p) } > .toMap > } > > > If you could give me a tipp there would be g

Re: Spark and Accumulo Delegation tokens

2018-03-23 Thread Saisai Shao
er) > > // Filter out providers for which > spark.security.credentials.{service}.enabled is false. > providers > .filter { p => isServiceEnabled(p.serviceName) } > .map { p => (p.serviceName, p) } > .toMap > } > > > If you could give me a tipp there would be g

Re: Spark and Accumulo Delegation tokens

2018-03-22 Thread Saisai Shao
I think you can build your own Accumulo credential provider as similar to HadoopDelegationTokenProvider out of Spark, Spark already provided an interface "ServiceCredentialProvider" for user to plug-in customized credential provider. Thanks Jerry 2018-03-23 14:29 GMT+08:00 Jorge Machado : > Hi G

Re: Spark and Accumulo Delegation tokens

2018-03-22 Thread Saisai Shao
I think you can build your own Accumulo credential provider as similar to HadoopDelegationTokenProvider out of Spark, Spark already provided an interface "ServiceCredentialProvider" for user to plug-in customized credential provider. Thanks Jerry 2018-03-23 14:29 GMT+08:00 Jorge Machado : > Hi G

[jira] [Resolved] (SPARK-23361) Driver restart fails if it happens after 7 days from app submission

2018-03-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23361. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20657 [https

[jira] [Assigned] (SPARK-23361) Driver restart fails if it happens after 7 days from app submission

2018-03-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23361: --- Assignee: Marcelo Vanzin > Driver restart fails if it happens after 7 days from

Re: Query on creating multiple livy sessions in parallel

2018-03-19 Thread Saisai Shao
This might be a BUG. If possible can you please create a JIRA to track this issue. Thanks! Best, Jerry 2018-03-19 20:18 GMT+08:00 Rao, Abhishek (Nokia - IN/Bangalore) < abhishek@nokia.com>: > Hi, > > We're trying to create multiple livy sessions in parallel and then using > them. But when we

[jira] [Updated] (SPARK-23644) SHS with proxy doesn't show applications

2018-03-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23644: Fix Version/s: 2.3.1 > SHS with proxy doesn't show appl

[jira] [Issue Comment Deleted] (SPARK-23534) Spark run on Hadoop 3.0.0

2018-03-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23534: Comment: was deleted (was: I'm curious why there's a library which depends on a such n

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

2018-03-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405659#comment-16405659 ] Saisai Shao commented on SPARK-23534: - I'm curious why there's a

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

2018-03-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405658#comment-16405658 ] Saisai Shao commented on SPARK-23534: - I'm curious why there's a

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

2018-03-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405649#comment-16405649 ] Saisai Shao commented on SPARK-23534: - I'm trying to move forward th

[jira] [Resolved] (SPARK-23708) Comment of ShutdownHookManager.addShutdownHook is error

2018-03-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23708. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20845 [https

[jira] [Assigned] (SPARK-23708) Comment of ShutdownHookManager.addShutdownHook is error

2018-03-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23708: --- Assignee: zhoukang > Comment of ShutdownHookManager.addShutdownHook is er

[jira] [Resolved] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

2018-03-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23635. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20799 [https

[jira] [Assigned] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

2018-03-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23635: --- Assignee: Saisai Shao > Spark executor env variable is overwritten by same name AM

[jira] [Commented] (SPARK-23644) SHS with proxy doesn't show applications

2018-03-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401538#comment-16401538 ] Saisai Shao commented on SPARK-23644: - Fixed in https://github.com/apache/spark/

[jira] [Resolved] (SPARK-23644) SHS with proxy doesn't show applications

2018-03-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23644. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0 > SHS with pr

Re: Development in Intellij IDEA

2018-03-14 Thread Saisai Shao
Like what Alex mentioned, due to the support of multi-version Scala in one project, we add some tricks in Maven to make it work. But unfortunately Intellij IDEA is not intelligent enough to recognize such tricks. So it is hard to build and run test in IDEA. Maybe you need to manually add something.

[jira] [Resolved] (LIVY-446) Support Spark 2.3 for Livy

2018-03-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-446. -- Resolution: Fixed Fix Version/s: 0.6.0 > Support Spark 2.3 for L

[jira] [Created] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

2018-03-08 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-23635: --- Summary: Spark executor env variable is overwritten by same name AM env variable Key: SPARK-23635 URL: https://issues.apache.org/jira/browse/SPARK-23635 Project: Spark

[jira] [Updated] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

2018-03-08 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23635: Description: In the current Spark on YARN code, AM always will copy and overwrite its env

[jira] [Resolved] (LIVY-448) Assembly format is duplicated in pom.xml and assembly.xml

2018-03-07 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-448. -- Resolution: Fixed > Assembly format is duplicated in pom.xml and assembly.

[jira] [Updated] (LIVY-448) Assembly format is duplicated in pom.xml and assembly.xml

2018-03-07 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-448: - Fix Version/s: (was: 0.5.1) > Assembly format is duplicated in pom.xml and assembly.

[jira] [Created] (LIVY-449) Livy to Support Hadoop 3.0.0

2018-03-07 Thread Saisai Shao (JIRA)
Saisai Shao created LIVY-449: Summary: Livy to Support Hadoop 3.0.0 Key: LIVY-449 URL: https://issues.apache.org/jira/browse/LIVY-449 Project: Livy Issue Type: Improvement Affects Versions

[jira] [Comment Edited] (SPARK-23534) Spark run on Hadoop 3.0.0

2018-03-06 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388761#comment-16388761 ] Saisai Shao edited comment on SPARK-23534 at 3/7/18 12:3

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

2018-03-06 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388761#comment-16388761 ] Saisai Shao commented on SPARK-23534: - I don't think so. Spark uses its

Fwd: [apache/incubator-livy] One of your dependencies may have a security vulnerability

2018-03-06 Thread Saisai Shao
Hi Alex, Would you please check again? Thanks! Best regards -- Forwarded message -- From: Apache Security Team Date: 2018-03-06 16:39 GMT+08:00 Subject: Re: [apache/incubator-livy] One of your dependencies may have a security vulnerability To: Saisai Shao Cc: priv

[jira] [Created] (LIVY-446) Support Spark 2.3 for Livy

2018-03-04 Thread Saisai Shao (JIRA)
Saisai Shao created LIVY-446: Summary: Support Spark 2.3 for Livy Key: LIVY-446 URL: https://issues.apache.org/jira/browse/LIVY-446 Project: Livy Issue Type: Improvement Components

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-03-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385539#comment-16385539 ] Saisai Shao commented on HIVE-16391: Hi all, Do we have any progress on it? S

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-03-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385474#comment-16385474 ] Saisai Shao commented on SPARK-18673: - Thanks for the info [~aihuaxu]. > Dat

Re: Welcoming some new committers

2018-03-02 Thread Saisai Shao
Congrats to everyone! Thanks Jerry 2018-03-03 15:30 GMT+08:00 Liang-Chi Hsieh : > > Congrats to everyone! > > > Kazuaki Ishizaki wrote > > Congratulations to everyone! > > > > Kazuaki Ishizaki > > > > > > > > From: Takeshi Yamamuro < > > > linguin.m.s@ > > > > > > To: Spark dev list < > >

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

2018-03-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382932#comment-16382932 ] Saisai Shao commented on SPARK-23534: - One issue is Hive 1.2.1.spark2 rejects Ha

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-03-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382929#comment-16382929 ] Saisai Shao commented on SPARK-18673: - Spark itself uses it own hive (1.2.1.sp

[jira] [Updated] (SPARK-23534) Spark run on Hadoop 3.0.0

2018-02-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23534: Description: Major Hadoop vendors already/will step in Hadoop 3.0. So we should also make sure

[jira] [Created] (SPARK-23534) Spark run on Hadoop 3.0.0

2018-02-28 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-23534: --- Summary: Spark run on Hadoop 3.0.0 Key: SPARK-23534 URL: https://issues.apache.org/jira/browse/SPARK-23534 Project: Spark Issue Type: Improvement

[jira] [Resolved] (LIVY-445) Upgrade Netty version to avoid security issue

2018-02-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-445. -- Resolution: Duplicate > Upgrade Netty version to avoid security is

[jira] [Closed] (LIVY-445) Upgrade Netty version to avoid security issue

2018-02-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed LIVY-445. > Upgrade Netty version to avoid security is

[jira] [Created] (LIVY-445) Upgrade Netty version to avoid security issue

2018-02-27 Thread Saisai Shao (JIRA)
Saisai Shao created LIVY-445: Summary: Upgrade Netty version to avoid security issue Key: LIVY-445 URL: https://issues.apache.org/jira/browse/LIVY-445 Project: Livy Issue Type: Improvement

Re: Error - HDP 2.6.4 - Livy 0.4.0-SNAPSHOT

2018-02-22 Thread Saisai Shao
Can you please try 0.5.0 release version ( http://livy.incubator.apache.org/download/). I assume 0.5.0 should have fixed this concurrent issue. Thanks Jerry 2018-02-23 14:42 GMT+08:00 Sudha KS : > Hi all, > > Anybody facing this error? > I get error running example code (PiJob) compiled on Scala

Re: Predef does not have a member classOf

2018-02-22 Thread Saisai Shao
This seems like a Scala version issue (as far as I can see from the stack). Would you please describe the details about how you use Livy and what configurations you set? Thanks Jerry 2018-02-23 6:30 GMT+08:00 David Espinosa : > Hi all, > > I'm a new user in Livy. I've created a scala app that ru

[jira] [Commented] (SPARK-12140) Support Streaming UI in HistoryServer

2018-02-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363300#comment-16363300 ] Saisai Shao commented on SPARK-12140: - [~gschiavon] the community has concern a

[jira] [Commented] (SPARK-23096) Migrate rate source to v2

2018-02-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358064#comment-16358064 ] Saisai Shao commented on SPARK-23096: - [~joseph.torres] [~tdas] can I take a c

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357824#comment-16357824 ] Saisai Shao commented on SPARK-23206: - [~cltlfcjin] from Ebay also plans t

Re: Is multiple languages support in 0.5.0 only work with rest-api?

2018-02-06 Thread Saisai Shao
ontext > > <https://livy.incubator.apache.org/docs/latest/api/scala/index.html#org.apache.livy.scalaapi.ScalaJobContext>refers > to the same concept of session in REST api? > > On Tue, Feb 6, 2018 at 3:22 PM, Saisai Shao > wrote: > >> IIUC, I assume you're using JobAPI rather than

Re: Is multiple languages support in 0.5.0 only work with rest-api?

2018-02-05 Thread Saisai Shao
IIUC, I assume you're using JobAPI rather than REST APIs. Yes, you can create a session using Java/Scala job API, and then manipulate with Python APIs. Thanks Jerry 2018-02-06 13:43 GMT+08:00 simsicon : > Hi, > > I am pretty new to livy, and I am looking into the livy to do some > research if li

Re: Podling Report Reminder - February 2018

2018-02-05 Thread Saisai Shao
Great, thanks Alex! Best regards, Jerry 2018-02-06 7:56 GMT+08:00 Alex Bozarth : > I've updated the report if a mentor could take a look and sign off (or > give feedback). > > > *Alex Bozarth* > Software Engineer > Spark Technology Center > -- > *E-mail:* *ajboz...@us

[jira] [Commented] (SPARK-23235) Add executor Threaddump to api

2018-02-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349604#comment-16349604 ] Saisai Shao commented on SPARK-23235: - There's a new similar Jira (SP

[jira] [Updated] (SPARK-23200) Reset configuration when restarting from checkpoints

2018-01-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23200: Fix Version/s: (was: 2.4.0) > Reset configuration when restarting from checkpoi

[jira] [Reopened] (SPARK-23200) Reset configuration when restarting from checkpoints

2018-01-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reopened SPARK-23200: - Assignee: (was: Santiago Saavedra) > Reset configuration when restarting from checkpoi

[jira] [Resolved] (SPARK-23279) Avoid triggering distributed job for Console sink

2018-01-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23279. - Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.3.0 > Avoid trigger

[jira] [Commented] (SPARK-23279) Avoid triggering distributed job for Console sink

2018-01-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346301#comment-16346301 ] Saisai Shao commented on SPARK-23279: - Issue resolved by pull request 20447 h

[jira] [Created] (SPARK-23279) Avoid triggering distributed job for Console sink

2018-01-30 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-23279: --- Summary: Avoid triggering distributed job for Console sink Key: SPARK-23279 URL: https://issues.apache.org/jira/browse/SPARK-23279 Project: Spark Issue Type

[jira] [Resolved] (SPARK-23088) History server not showing incomplete/running applications

2018-01-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23088. - Resolution: Fixed Assignee: paul mackles Fix Version/s: 2.4.0 Issue resolved by

Re: [VOTE] Livy 0.5.0-incubating (RC2)

2018-01-29 Thread Saisai Shao
; San Francisco, CA 94105 > <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States&entry=gmail&source=g> > United States > <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States&entry=gmail&source=g> >

Re: [VOTE] Livy 0.5.0-incubating (RC2)

2018-01-29 Thread Saisai Shao
Hi Alex, Did you push artifacts to staging repo? I think I mentioned before to push artifacts to staging repo when you initiate a new RC vote. 2018-01-30 10:21 GMT+08:00 Alex Bozarth : > The votes passes with three +1s and a -0. I will be sending an email to > the Incubator PMC for a vote soon.

Re: In POST /sessions, what's the difference between "jars" and "spark.jar" in conf ?

2018-01-28 Thread Saisai Shao
I think they are basically the same, you can use either. Why don't you take a try. Thanks Jerry 2018-01-29 12:37 GMT+08:00 Kyunam Kim : > Basically, which option is equivalent to "spark-submit --jar > myLib0,jar,myLib1.jar" ? > > Thanks, > Q >

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

2018-01-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16342824#comment-16342824 ] Saisai Shao commented on SPARK-23187: - I think I tried, {{accumUpdates}} ca

Re: [VOTE] Livy 0.5.0-incubating (RC2)

2018-01-26 Thread Saisai Shao
Ideally, when a new RC is out, we should put release package on svn, and push artifacts to staging repo. Once vote is passed, just clicking “releas” to publish to maven repo is enough. We can push RC2 artifacts to staging repo for now. Alex Bozarth 于2018年1月27日 周六上午6:32写道: > Just wondering, but s

[jira] [Created] (SPARK-23228) Able to track Python create SparkSession in JVM

2018-01-26 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-23228: --- Summary: Able to track Python create SparkSession in JVM Key: SPARK-23228 URL: https://issues.apache.org/jira/browse/SPARK-23228 Project: Spark Issue Type

[jira] [Assigned] (SPARK-23200) Reset configuration when restarting from checkpoints

2018-01-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23200: --- Assignee: Anirudh Ramanathan > Reset configuration when restarting from checkpoi

[jira] [Commented] (SPARK-23200) Reset configuration when restarting from checkpoints

2018-01-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340648#comment-16340648 ] Saisai Shao commented on SPARK-23200: - Issue resolved by pull request 20383 h

[jira] [Resolved] (SPARK-23200) Reset configuration when restarting from checkpoints

2018-01-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23200. - Resolution: Fixed Fix Version/s: 2.4.0 > Reset configuration when restarting f

Re: [VOTE] Livy 0.5.0-incubating (RC2)

2018-01-25 Thread Saisai Shao
asc, sha512 and md5 LGTM. Also checked the package, seems everything is landed. Alex, did you also publish Livy artifacts to staging repo? 2018-01-26 8:03 GMT+08:00 Alex Bozarth : > That works for me, voting will be open an extra day for a total of 96 > hours ending on Tuesday January 30th at 12

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-01-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340536#comment-16340536 ] Saisai Shao commented on SPARK-23206: - Would you please summarize what kin

[jira] [Resolved] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

2018-01-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23187. - Resolution: Not A Problem > Accumulator object can not be sent from Executor to Dri

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

2018-01-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340522#comment-16340522 ] Saisai Shao commented on SPARK-23187: - I'm going to close this JIRA, as t

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

2018-01-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340521#comment-16340521 ] Saisai Shao commented on SPARK-23187: - I'm going to close this JIRA, as t

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

2018-01-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338737#comment-16338737 ] Saisai Shao commented on SPARK-23187: - Actually heartbeat report is OK accordin

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-01-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338553#comment-16338553 ] Saisai Shao commented on SPARK-23206: - I think this Jira duplicates SPARK-9103.

Fwd: [apache/incubator-livy] One of your dependencies may have a security vulnerability

2018-01-23 Thread Saisai Shao
Hi Alex, Is it due to your recent changes to add ruby file? Thanks Jerry -- Forwarded message -- From: Greg Stein Date: 2018-01-24 6:14 GMT+08:00 Subject: Fwd: [apache/incubator-livy] One of your dependencies may have a security vulnerability To: priv...@livy.incubator.apache.or

<    2   3   4   5   6   7   8   9   10   11   >