Re: Fwd: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

Thanks Alex for your check. I'm not sure if we have a proper place to add
this reference in the document. I'm fine with the current status, and will
add the comment during release vote.

Best regards,
Saisai

Alex Bozarth  于2020年1月7日周二 上午4:04写道：

> On the note of the "missing" licenses:
>
> Since I added those I did a bit of research to refresh my memory. Based on
> https://www.glyphicons.com/license/ we do not need to directly mention
> the licensing for those fonts. This is again backed up by the comment at
> the top of the bootstrap page
> https://getbootstrap.com/docs/3.3/components/ which is where our usage of
> the fonts/icons comes from. Based on these links we are fine as is but
> could also add a mention somewhere in our docs/licenses to the glyphicons
> website. Another reference would be this PR for the retired Apache project
> Falcon which also used these fonts
> https://issues.apache.org/jira/browse/FALCON-2110
>
> tl:dr we're fine as we are, but it might be kind to add a reference
> somewhere in our docs
>
>
> *Alex Bozarth*
> Software Engineer
> Center for Open-Source Data & AI Technologies
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Saisai Shao ---01/05/2020 10:22:40
> PM---Does anyone knows how to handle license issue for fonts like]Saisai
> Shao ---01/05/2020 10:22:40 PM---Does anyone knows how to handle license
> issue for fonts like below? Thanks
>
> From: Saisai Shao 
> To: dev@livy.incubator.apache.org
> Date: 01/05/2020 10:22 PM
> Subject: [EXTERNAL] Fwd: [VOTE] Release Apache Livy 0.7.0 (incubating)
> based on RC3
>
> --
>
>
>
> Does anyone knows how to handle license issue for fonts like below?
>
> Thanks
> Saisai
>
> -- Forwarded message -
> 发件人： Justin Mclean 
> Date: 2020年1月3日周五 上午9:59
> Subject: Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3
> To: 
>
>
> Hi,
>
> +1 (binding). There’s a license issue that needs to be fixed before the
> next release.
>
> I checked:
> - incubating in name
> - DISCLAIMER exists
> - LICENSE is missing a few things
> - NOTICE year needs updating (Is it still 2018?)
> - No unexpended binary files
> - all source files have ASF headers
> - can compile from source
>
> License is missing license for [1][2]
>
> Thanks,
> Justin
>
> 1.
>
> ./docs/assets/themes/apache/bootstrap/fonts/glyphicons-halflings-regular.svg
> 2.
>
> ./server/src/main/resources/org/apache/livy/server/ui/static/fonts/glyphicons-halflings-regular.*
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>
>
>

Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

Hi Luciano,

I've fixed the license version manually, they'll be solved in next RC cut.
But still I cannot find out the license for ldapsdk 4.1, this is a very old
library, I doubt there's no license existed. Would you please help to check
it? Thanks!

And regarding to bootstrap and them files, they do existed in binary distro
as a part of Livy Server.

Thanks
Saisai

Luciano Resende  于2020年1月3日周五 上午6:11写道：

> On Mon, Dec 30, 2019 at 3:43 AM Saisai Shao 
> wrote:
> >
> > The Livy PPMC has voted to release Livy 0.7.0 RC3 as the next Livy
> release.
> >
> > Livy enables programmatic, fault-tolerant, multi-tenant submission of
> > Spark jobs from web/mobile apps (no Spark client needed). So, multiple
> > users can interact with your Spark cluster concurrently and reliably.
> >
> > Vote thread:
> >
> https://lists.apache.org/thread.html/ce78a69fb17b7ae1a04a51cb13949c4f726d03c43c6ebde8bbd0272f%40%3Cdev.livy.apache.org%3E
> >
> > Result thread:
> >
> https://lists.apache.org/thread.html/d030023995e4323c32e8ee68de99e48cb4f20761d5ff784ff2aac558%40%3Cdev.livy.apache.org%3E
> >
> > The RC is based on tag v0.7.0-incubating-rc3:
> >
> https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da
> >
> > The release files can be found here:
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/
> >
> > The staged maven artifacts can be found here:
> > https://repository.apache.org/content/repositories/orgapachelivy-1012
> >
> > The list of resolved JIRAs in this release can be found here:
> > https://issues.apache.org/jira/projects/LIVY/versions/12345179
> >
> > Vote will be open for at least 72 hours. Thanks!
>
> While looking into the binary distribution, it looks like the
> third-party file might have been generated and has more files that
> actually the ones being distributed, but my bigger issue is with a
> list of dependencies with unknown licenses.
>
> Unknown license:
>
> * ASM Core (asm:asm:3.1 - http://asm.objectweb.org/asm/)
> * Antlr 3.4 Runtime (org.antlr:antlr-runtime:3.4 -
> http://www.antlr.org)
> * Java Transaction API (javax.transaction:jta:1.1 -
> http://java.sun.com/products/jta)
> * Jettison (org.codehaus.jettison:jettison:1.1 - no url defined)
> * commons-beanutils (commons-beanutils:commons-beanutils:1.7.0 -
> no url defined)
> * jsp-api (javax.servlet:jsp-api:2.0 - no url defined)
> * ldapsdk (ldapsdk:ldapsdk:4.1 - no url defined)
> * oro (oro:oro:2.0.8 - no url defined)
> * servlet-api (javax.servlet:servlet-api:2.5 - no url defined)
> * transaction-api (javax.transaction:transaction-api:1.1 - no url
> defined)
> * zookeeper (org.apache.zookeeper:zookeeper:3.4.6 - no url defined)
>
> Also, the main license lists dependencies (e.g. bootstrap and related
> theme files) that I believe is only in source distro.
>
> I would feel more comfortable approving the release after this is
> sorted out, or making this a source release only.
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

[jira] [Created] (LIVY-738) Fix Livy third-party library license generating issue

[Saisai Shao (Jira)]

Saisai Shao created LIVY-738:


 Summary: Fix Livy third-party library license generating issue
 Key: LIVY-738
 URL: https://issues.apache.org/jira/browse/LIVY-738
 Project: Livy
  Issue Type: Bug
  Components: Build
Affects Versions: 0.8.0
Reporter: Saisai Shao


Some dependencies' licenses are missing, so it cannot correctly generate the 
license information, here propose to fix them manually.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Fwd: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

Does anyone knows how to handle license issue for fonts like below?

Thanks
Saisai

-- Forwarded message -
发件人： Justin Mclean 
Date: 2020年1月3日周五 上午9:59
Subject: Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3
To: 


Hi,

+1 (binding). There’s a license issue that needs to be fixed before the
next release.

I checked:
- incubating in name
- DISCLAIMER exists
- LICENSE is missing a few things
- NOTICE year needs updating (Is it still 2018?)
- No unexpended binary files
- all source files have ASF headers
- can compile from source

License is missing license for [1][2]

Thanks,
Justin

1.
./docs/assets/themes/apache/bootstrap/fonts/glyphicons-halflings-regular.svg
2.
./server/src/main/resources/org/apache/livy/server/ui/static/fonts/glyphicons-halflings-regular.*


-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Podling Tubemq Report Reminder - January 2020

[Saisai Shao]

I've updated some parts of the report, please help to review, thanks!

Besides, we will follow the Apache way to bring out all the discussions on
the mail list, thanks for your guidance.

Thanks
Saisai


Saisai Shao  于2020年1月3日周五 上午11:27写道：

> Thanks Justin for your reply, I will amend the report soon.
>
> Best regards,
> Saisai
>
> Justin Mclean  于2020年1月3日周五 上午7:03写道：
>
>> Hi,
>>
>> Thanks for submitting the  report but it will not be accepted in it's
>> currently form as it doesn't accurately reflect the current podlings state
>> and what it needs to do. Next time I suggest you work on the report in the
>> open before the due date.
>>
>> You have user the three most important unfinished issues to address
>> before graduating:
>> 1. Grow the community to involve more contributors and increase the
>> diversity.
>> 2. Polish the code and document to satisfy the Apache way.
>> 3. Publish the Apache release.
>>
>> There are far more serious matters that need to be sorted out before
>> those are addressed. Number one being moving conversation to this mailing
>> list.
>>
>> Under "Are there any issues that the IPMC or ASF Board need to be aware
>> of?" you have said "No" this is incorrect.
>>
>> Under "How has the community developed since the last report?" you
>> mention PRs and commits but the repos have not moved to the ASF yet so how
>> is this possible?
>>
>> User "How has the project developed since the last report?" you mention
>> working on the first release and website. I see no conversation of this on
>> list or any work towards those gaols. http://tubemq.apache.org currently
>> gives 404 not found.
>>
>> Please rewrite the report to accurately reflect where the podling
>> currently is and what issues it faces.
>>
>> Thanks,
>> Justin
>>
>>
>>
>>
>>
>>

Re: Podling Tubemq Report Reminder - January 2020

[Saisai Shao]

Thanks Justin for your reply, I will amend the report soon.

Best regards,
Saisai

Justin Mclean  于2020年1月3日周五 上午7:03写道：

> Hi,
>
> Thanks for submitting the  report but it will not be accepted in it's
> currently form as it doesn't accurately reflect the current podlings state
> and what it needs to do. Next time I suggest you work on the report in the
> open before the due date.
>
> You have user the three most important unfinished issues to address before
> graduating:
> 1. Grow the community to involve more contributors and increase the
> diversity.
> 2. Polish the code and document to satisfy the Apache way.
> 3. Publish the Apache release.
>
> There are far more serious matters that need to be sorted out before those
> are addressed. Number one being moving conversation to this mailing list.
>
> Under "Are there any issues that the IPMC or ASF Board need to be aware
> of?" you have said "No" this is incorrect.
>
> Under "How has the community developed since the last report?" you mention
> PRs and commits but the repos have not moved to the ASF yet so how is this
> possible?
>
> User "How has the project developed since the last report?" you mention
> working on the first release and website. I see no conversation of this on
> list or any work towards those gaols. http://tubemq.apache.org currently
> gives 404 not found.
>
> Please rewrite the report to accurately reflect where the podling
> currently is and what issues it faces.
>
> Thanks,
> Justin
>
>
>
>
>
>

Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

Thanks Justin and Luciano for your check.

I use some maven plugins to extract license files from third party
libraries, the missing of license files are mainly due to the issue of this
jars. I will amend them manually and make another RC release.

Thanks
Saisai



Justin Mclean  于2020年1月3日周五 上午9:59写道：

> Hi,
>
> +1 (binding). There’s a license issue that needs to be fixed before the
> next release.
>
> I checked:
> - incubating in name
> - DISCLAIMER exists
> - LICENSE is missing a few things
> - NOTICE year needs updating (Is it still 2018?)
> - No unexpended binary files
> - all source files have ASF headers
> - can compile from source
>
> License is missing license for [1][2]
>
> Thanks,
> Justin
>
> 1.
> ./docs/assets/themes/apache/bootstrap/fonts/glyphicons-halflings-regular.svg
> 2.
> ./server/src/main/resources/org/apache/livy/server/ui/static/fonts/glyphicons-halflings-regular.*
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

There's no any vote yet, I guess it might be due to the holiday season. I
will extend the due date to weekend.

Thanks
Saisai

Saisai Shao  于2019年12月30日周一 上午10:43写道：

> The Livy PPMC has voted to release Livy 0.7.0 RC3 as the next Livy
>  release.
>
> Livy enables programmatic, fault-tolerant, multi-tenant submission of
> Spark jobs from web/mobile apps (no Spark client needed). So, multiple
> users can interact with your Spark cluster concurrently and reliably.
>
> Vote thread:
>
> https://lists.apache.org/thread.html/ce78a69fb17b7ae1a04a51cb13949c4f726d03c43c6ebde8bbd0272f%40%3Cdev.livy.apache.org%3E
>
> Result thread:
>
> https://lists.apache.org/thread.html/d030023995e4323c32e8ee68de99e48cb4f20761d5ff784ff2aac558%40%3Cdev.livy.apache.org%3E
>
> The RC is based on tag v0.7.0-incubating-rc3:
>
> https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da
>
> The release files can be found here:
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/
>
> The staged maven artifacts can be found here:
> https://repository.apache.org/content/repositories/orgapachelivy-1012
>
> The list of resolved JIRAs in this release can be found here:
> https://issues.apache.org/jira/projects/LIVY/versions/12345179
>
> Vote will be open for at least 72 hours. Thanks!
>

Re: Podling Tubemq Report Reminder - January 2020

[Saisai Shao]

Hi Justin,

Sorry about the late reply, we will work on this today. Thanks a lot for
your reminder.

Best regards,
Saisai

Justin Mclean  于2020年1月2日周四 上午10:05写道：

> Hi,
>
> This is the second report in a row the project has missed, one missed
> report is OK, two are cause for concern, three and you may be kicked out of
> the Incubator. I can see that you are still moving resources to the
> incubator and start up here has been slow. You need to provide the IPMC
> with oversight to how the project is going and what issues it is facing and
> still need to submit reports in the start up phase.
>
> It seems to me that some of your mentors are missing, if this is the case
> please mention this in your report. I've reached out to them (on the
> private list) and if they don't reply they will be removed and other
> mentors found to replace them.
>
> I suggest you discuss on this list what is need to move the project over,
> I'm not seeing a lot of communication on these lists, and it seems that
> some communication is currently occurring elsewhere. Please fix this.
>
> Thanks,
> Justin
>

Re: Podling Livy Report Reminder - January 2020

[Saisai Shao]

I will work on this.

Thanks
Saisai

Justin Mclean  于2020年1月1日周三 下午4:06写道：

> Hi,
> Just a friendly reminder the report is due today is anyone working on it?
> Thanks,
> Justin
>

[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17005895#comment-17005895
 ] 

Saisai Shao commented on LIVY-718:
--

IIUC, current JDBC part maintains lots of results/metadata/information on 
LivyServer, I think the key point mentioned by [~bikassaha] is to make Livy 
Server stateless, that would be an ideal solution, but currently it requires a 
large amount of works.

> Support multi-active high availability in Livy
> --
>
> Key: LIVY-718
> URL: https://issues.apache.org/jira/browse/LIVY-718
> Project: Livy
>  Issue Type: Epic
>  Components: RSC, Server
>Reporter: Yiheng Wang
>Priority: Major
>
> In this JIRA we want to discuss how to implement multi-active high 
> availability in Livy.
> Currently, Livy only supports single node recovery. This is not sufficient in 
> some production environments. In our scenario, the Livy server serves many 
> notebook and JDBC services. We want to make Livy service more fault-tolerant 
> and scalable.
> There're already some proposals in the community for high availability. But 
> they're not so complete or just for active-standby high availability. So we 
> propose a multi-active high availability design to achieve the following 
> goals:
> # One or more servers will serve the client requests at the same time.
> # Sessions are allocated among different servers.
> # When one node crashes, the affected sessions will be moved to other active 
> services.
> Here's our design document, please review and comment:
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] Release Apache Livy 0.7.0 (incubating) based on RC3

[Saisai Shao]

The Livy PPMC has voted to release Livy 0.7.0 RC3 as the next Livy release.

Livy enables programmatic, fault-tolerant, multi-tenant submission of
Spark jobs from web/mobile apps (no Spark client needed). So, multiple
users can interact with your Spark cluster concurrently and reliably.

Vote thread:
https://lists.apache.org/thread.html/ce78a69fb17b7ae1a04a51cb13949c4f726d03c43c6ebde8bbd0272f%40%3Cdev.livy.apache.org%3E

Result thread:
https://lists.apache.org/thread.html/d030023995e4323c32e8ee68de99e48cb4f20761d5ff784ff2aac558%40%3Cdev.livy.apache.org%3E

The RC is based on tag v0.7.0-incubating-rc3:
https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da

The release files can be found here:
https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/

The staged maven artifacts can be found here:
https://repository.apache.org/content/repositories/orgapachelivy-1012

The list of resolved JIRAs in this release can be found here:
https://issues.apache.org/jira/projects/LIVY/versions/12345179

Vote will be open for at least 72 hours. Thanks!

Re: [VOTE] Release Livy 0.7.0 based on RC3

[Saisai Shao]

We have 3 binding +1, no 0s or -1s. Vote passes, I'll follow up on the
general@ list.

Thanks
Saisai

Bikas Saha  于2019年12月29日周日 上午11:25写道：

> +1
>
> Bikas
>
> 
> From: Aliaksandr Sasnouskikh 
> Sent: Monday, December 23, 2019 11:58 PM
> To: dev@livy.incubator.apache.org 
> Subject: Re: [VOTE] Release Livy 0.7.0 based on RC3
>
> +1
>
> вт, 24 дек. 2019 г. в 8:11, Saisai Shao :
>
> > The date is overdue, but we still didn't receive enough votes, let me
> > extend the due date to Dec 29, 23:59 UTC. Please help to vote.
> >
> > Best regards,
> > Saisai
> >
> > Saisai Shao  于2019年12月20日周五 下午4:02写道：
> >
> > > +1 myself.
> > >
> > > Yiheng Wang  于2019年12月19日周四 上午11:41写道：
> > >
> > >> Check the release files. Looks good.
> > >>
> > >> +1
> > >>
> > >> On Wed, Dec 18, 2019 at 5:25 PM runzhiwang 
> > >> wrote:
> > >>
> > >> >  
> > >> >   +1
> > >> >   
> > >> >  ** release files*
> > >> > I verify the apache-livy-0.7.0-incubating-bin.zip by creating
> session,
> > >> > statement, and everything is ok.
> > >> >  I also verify the sha512 and asc file. Both look good.
> > >> >
> > >> >   ** staged maven artifacts **
> > >> >  It's ok.
> > >> >   
> > >> >
> > >> >  ** resolved JIRAs **
> > >> >  It's ok. I check all the jiras in the commits.
> > >> >   
> > >> >
> > >> > Thanks
> > >> > RunzhiWang
> > >> >
> > >> >
> > >> >
> > >> >  -- 原始邮件 --
> > >> >   发件人: "Saisai Shao" > >> >  发送时间: 2019年12月18日(星期三) 中午1:58
> > >> >  收件人: "dev" > >> >
> > >> >  主题: [VOTE] Release Livy 0.7.0 based on RC3
> > >> >
> > >> >
> > >> >
> > >> > This vote is for releasing Livy 0.7.0 based on RC3.
> > >> >
> > >> > The vote will be open until Friday  Dec 23, 23:59 UTC and
> > >> > will pass with a minimum of 3 +1 binding votes and a majority of
> > >> positive
> > >> > votes.
> > >> >
> > >> > [+1] This release is ready to send to the Incubator PMC for approval
> > >> >
> > >> > [-1] This release is not ready because...
> > >> >
> > >> > This vote is held according to Apache Incubator release policy:
> > >> > https://incubator.apache.org/policy/incubation.html#releases
> > >> >
> > >> > The RC is based on tag v0.7.0-incubating-rc3:
> > >> >
> > >> >
> > >>
> >
> https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da
> > >> >
> > >> > The release files can be found here:
> > >> >
> > >>
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/
> > >> >
> > >> > The staged maven artifacts can be found here:
> > >> >
> https://repository.apache.org/content/repositories/orgapachelivy-1012
> > >> >
> > >> > The list of resolved JIRAs in this release can be found here:
> > >> > https://issues.apache.org/jira/projects/LIVY/versions/12345179
> > >> >
> > >> > Thanks
> > >> > Saisai
> > >>
> > >
> >
> --
> Отправлено с мобильного устройства
>

Re: [VOTE] Release Livy 0.7.0 based on RC3

[Saisai Shao]

The date is overdue, but we still didn't receive enough votes, let me
extend the due date to Dec 29, 23:59 UTC. Please help to vote.

Best regards,
Saisai

Saisai Shao  于2019年12月20日周五 下午4:02写道：

> +1 myself.
>
> Yiheng Wang  于2019年12月19日周四 上午11:41写道：
>
>> Check the release files. Looks good.
>>
>> +1
>>
>> On Wed, Dec 18, 2019 at 5:25 PM runzhiwang 
>> wrote:
>>
>> >  
>> >   +1
>> >   
>> >  ** release files*
>> > I verify the apache-livy-0.7.0-incubating-bin.zip by creating session,
>> > statement, and everything is ok.
>> >  I also verify the sha512 and asc file. Both look good.
>> >
>> >   ** staged maven artifacts **
>> >  It's ok.
>> >   
>> >
>> >  ** resolved JIRAs **
>> >  It's ok. I check all the jiras in the commits.
>> >   
>> >
>> > Thanks
>> > RunzhiWang
>> >
>> >
>> >
>> >  -- 原始邮件 --
>> >   发件人: "Saisai Shao"> >  发送时间: 2019年12月18日(星期三) 中午1:58
>> >  收件人: "dev"> >
>> >  主题: [VOTE] Release Livy 0.7.0 based on RC3
>> >
>> >
>> >
>> > This vote is for releasing Livy 0.7.0 based on RC3.
>> >
>> > The vote will be open until Friday  Dec 23, 23:59 UTC and
>> > will pass with a minimum of 3 +1 binding votes and a majority of
>> positive
>> > votes.
>> >
>> > [+1] This release is ready to send to the Incubator PMC for approval
>> >
>> > [-1] This release is not ready because...
>> >
>> > This vote is held according to Apache Incubator release policy:
>> > https://incubator.apache.org/policy/incubation.html#releases
>> >
>> > The RC is based on tag v0.7.0-incubating-rc3:
>> >
>> >
>> https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da
>> >
>> > The release files can be found here:
>> >
>> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/
>> >
>> > The staged maven artifacts can be found here:
>> > https://repository.apache.org/content/repositories/orgapachelivy-1012
>> >
>> > The list of resolved JIRAs in this release can be found here:
>> > https://issues.apache.org/jira/projects/LIVY/versions/12345179
>> >
>> > Thanks
>> > Saisai
>>
>

Re: [VOTE] Release Livy 0.7.0 based on RC3

[Saisai Shao]

+1 myself.

Yiheng Wang  于2019年12月19日周四 上午11:41写道：

> Check the release files. Looks good.
>
> +1
>
> On Wed, Dec 18, 2019 at 5:25 PM runzhiwang 
> wrote:
>
> >  
> >   +1
> >   
> >  ** release files*
> > I verify the apache-livy-0.7.0-incubating-bin.zip by creating session,
> > statement, and everything is ok.
> >  I also verify the sha512 and asc file. Both look good.
> >
> >   ** staged maven artifacts **
> >  It's ok.
> >   
> >
> >  ** resolved JIRAs **
> >  It's ok. I check all the jiras in the commits.
> >   
> >
> > Thanks
> > RunzhiWang
> >
> >
> >
> >  -- 原始邮件 --
> >   发件人: "Saisai Shao" >  发送时间: 2019年12月18日(星期三) 中午1:58
> >  收件人: "dev" >
> >  主题: [VOTE] Release Livy 0.7.0 based on RC3
> >
> >
> >
> > This vote is for releasing Livy 0.7.0 based on RC3.
> >
> > The vote will be open until Friday  Dec 23, 23:59 UTC and
> > will pass with a minimum of 3 +1 binding votes and a majority of positive
> > votes.
> >
> > [+1] This release is ready to send to the Incubator PMC for approval
> >
> > [-1] This release is not ready because...
> >
> > This vote is held according to Apache Incubator release policy:
> > https://incubator.apache.org/policy/incubation.html#releases
> >
> > The RC is based on tag v0.7.0-incubating-rc3:
> >
> >
> https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da
> >
> > The release files can be found here:
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/
> >
> > The staged maven artifacts can be found here:
> > https://repository.apache.org/content/repositories/orgapachelivy-1012
> >
> > The list of resolved JIRAs in this release can be found here:
> > https://issues.apache.org/jira/projects/LIVY/versions/12345179
> >
> > Thanks
> > Saisai
>

[jira] [Updated] (LIVY-727) Session state always be idle though the yarn application has been killed after restart livy.

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated LIVY-727:
-
Affects Version/s: 0.7.0
   0.6.0

> Session state always be idle though the yarn application has been killed 
> after  restart livy.
> -
>
> Key: LIVY-727
> URL: https://issues.apache.org/jira/browse/LIVY-727
> Project: Livy
>  Issue Type: Bug
>Affects Versions: 0.6.0, 0.7.0
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.8.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> # Set livy.server.recovery.mode=recovery, and create a session in yarn-cluster
>  # Restart livy,then kill the yarn application of the session.
>  # The session state will always be idle and never change to killed or dead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-727) Session state always be idle though the yarn application has been killed after restart livy.

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-727.
--
Fix Version/s: 0.8.0
 Assignee: runzhiwang
   Resolution: Fixed

Issue resolved by pull request 272
https://github.com/apache/incubator-livy/pull/272

> Session state always be idle though the yarn application has been killed 
> after  restart livy.
> -
>
> Key: LIVY-727
> URL: https://issues.apache.org/jira/browse/LIVY-727
> Project: Livy
>  Issue Type: Bug
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.8.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> # Set livy.server.recovery.mode=recovery, and create a session in yarn-cluster
>  # Restart livy,then kill the yarn application of the session.
>  # The session state will always be idle and never change to killed or dead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-729) Livy should not recover the killed session

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-729.
--
Fix Version/s: 0.8.0
 Assignee: runzhiwang
   Resolution: Fixed

Issue resolved by pull request 266
https://github.com/apache/incubator-livy/pull/266

> Livy should not recover the killed session
> --
>
> Key: LIVY-729
> URL: https://issues.apache.org/jira/browse/LIVY-729
> Project: Livy
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 0.6.0
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.8.0
>
> Attachments: image-2019-12-18-08-56-46-925.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Follows are steps to reproduce the problem:
>  # Set livy.server.recovery.mode=recovery, and create a session: session0 in 
> yarn-cluster
>  # kill the yarn application of the session
>  # restart livy
>  # livy try to recover session0, but application has been killed and driver 
> does not exist, so client can not connect to driver, and exception was thrown 
> as the image.
>  # If the ip:port of the driver was reused by session1, client of session0 
> will try to connect to driver of session1, then driver will throw exception: 
> Unexpected client ID.
>  # Both the exception threw by livy and driver will confused the user, and 
> recover a lot of killed sessions will delay the recover of alive session.
> !image-2019-12-18-08-56-46-925.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] Release Livy 0.7.0 based on RC3

[Saisai Shao]

This vote is for releasing Livy 0.7.0 based on RC3.

The vote will be open until Friday  Dec 23, 23:59 UTC and
will pass with a minimum of 3 +1 binding votes and a majority of positive
votes.

[+1] This release is ready to send to the Incubator PMC for approval

[-1] This release is not ready because...

This vote is held according to Apache Incubator release policy:
https://incubator.apache.org/policy/incubation.html#releases

The RC is based on tag v0.7.0-incubating-rc3:
https://github.com/apache/incubator-livy/commit/0cc431a7e9236e313f9904ef121cad5df444d4da

The release files can be found here:
https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc3/

The staged maven artifacts can be found here:
https://repository.apache.org/content/repositories/orgapachelivy-1012

The list of resolved JIRAs in this release can be found here:
https://issues.apache.org/jira/projects/LIVY/versions/12345179

Thanks
Saisai

Re: [VOTE] Release Livy 0.7.0 based on RC2

[Saisai Shao]

Thanks guys,

I didn't add thrift profile into building, so the thrift related part are
missing. For 2.10 related poms/jars, because we removed the support of
2.10, so they're not existed anymore, also for minicluster related profile.

So this vote is failed, and I will start a new build with thrift included.

Thanks
Saisai


Yiheng Wang  于2019年12月16日周一 下午8:32写道：

> Hi
>
> ** maven staging artifacts*
> Compare the staging maven artifacts with 0.6.0, looks like some folders are
> missing:
> livy-beeline/
> livy-core_2.10/
> livy-repl_2.10/
> livy-scala-api_2.10/
> livy-thriftserver/
> livy-thriftserver-session/
> minicluster-dependencies-parent/
> minicluster-dependencies_2.10/
> minicluster-dependencies_2.11/
>
> ** release files*
> I verify the sha512 and asc file. Both look good.
>
> Thanks
> Yiheng
>
> On Mon, Dec 16, 2019 at 4:07 PM Saisai Shao 
> wrote:
>
> > This vote is for releasing Livy 0.7.0 based on RC2 (RC1 has one
> unnecessary
> > commit due to my bad).
> >
> > The vote will be open until Friday  Dec 20, 23:59 UTC and
> > will pass with a minimum of 3 +1 binding votes and a majority of positive
> > votes.
> >
> > [+1] This release is ready to send to the Incubator PMC for approval
> >
> > [-1] This release is not ready because...
> >
> > This vote is held according to Apache Incubator release policy:
> > https://incubator.apache.org/policy/incubation.html#releases
> >
> > The RC is based on tag v0.7.0-incubating-rc2:
> >
> >
> https://github.com/apache/incubator-livy/commit/eebb6ecfe5ec021244c69096fa240e84f933b2ca
> >
> > The release files can be found here:
> >
> https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc2/
> >
> > The staged maven artifacts can be found here:
> > https://repository.apache.org/content/repositories/orgapachelivy-1009
> >
> > The list of resolved JIRAs in this release can be found here:
> > https://issues.apache.org/jira/projects/LIVY/versions/12345179
> >
> > Thanks
> > Saisai
> >
>

[VOTE] Release Livy 0.7.0 based on RC2

[Saisai Shao]

This vote is for releasing Livy 0.7.0 based on RC2 (RC1 has one unnecessary
commit due to my bad).

The vote will be open until Friday  Dec 20, 23:59 UTC and
will pass with a minimum of 3 +1 binding votes and a majority of positive
votes.

[+1] This release is ready to send to the Incubator PMC for approval

[-1] This release is not ready because...

This vote is held according to Apache Incubator release policy:
https://incubator.apache.org/policy/incubation.html#releases

The RC is based on tag v0.7.0-incubating-rc2:
https://github.com/apache/incubator-livy/commit/eebb6ecfe5ec021244c69096fa240e84f933b2ca

The release files can be found here:
https://dist.apache.org/repos/dist/dev/incubator/livy/0.7.0-incubating-rc2/

The staged maven artifacts can be found here:
https://repository.apache.org/content/repositories/orgapachelivy-1009

The list of resolved JIRAs in this release can be found here:
https://issues.apache.org/jira/projects/LIVY/versions/12345179

Thanks
Saisai

Re: [Vote][LIVY-718] Support multi-active high availability in Livy

[Saisai Shao]

+1

rei sivan  于2019年12月12日周四 下午2:09写道：

> +1
>
> On Thu, Dec 12, 2019, 05:30 Yiheng Wang  wrote:
>
> > Dear Community
> >
> > I'd like to call a vote on LIVY-718
> >  "Support
> > multi-active high availability in Livy".
> >
> > Currently, Livy only supports single node recovery. This is not
> sufficient
> > in some production environments. In our scenario, the Livy server serves
> > many notebook and JDBC services. We want to make Livy service more
> > fault-tolerant and scalable.
> >
> > There're already some proposals in the community for high availability.
> But
> > they're not so complete or just for active-standby high availability. So
> we
> > propose a multi-active high availability design to achieve the following
> > goals:
> >
> >- One or more servers will serve the client requests at the same time.
> >- Sessions are allocated among different servers.
> >- When one node crashes, the affected sessions will be moved to other
> >active services.
> >
> > Please find the design doc here
> > <
> >
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
> > >,
> > which has been reviewed for several weeks.
> >
> > This vote is open until next Wednesday (Dec. 18).
> >
> > [] +1: Accept the proposal
> > [] +0
> > [] -1: I don't think this is a good idea because ...
> >
> > Thank you
> >
> > Yiheng
> >
>

Re: Support multi-active high availability in Livy

[Saisai Shao]

As I remembered we don't have such requirement in Livy community, but I'm
OK to start a vote. Yiheng, would you please start a vote?

Thanks
Saisai

Marco Gaido  于2019年12月12日周四 上午2:50写道：

> Shall we start an official vote before creating the subtasks?
>
> Thanks,
> Marco
>
> Il giorno mer 11 dic 2019 alle ore 14:50 Saisai Shao <
> sai.sai.s...@gmail.com>
> ha scritto:
>
> > I think it is feasible to move to next step. Let's create subtasks to
> track
> > all the works.
> >
> > Thanks
> > Saisai
> >
> > Yiheng Wang  于2019年12月10日周二 下午3:18写道：
> >
> > > Hi Community
> > >
> > > We have received some feedback for this design. Here're the open
> > questions
> > > for discussion
> > > 1. Session allocation algorithm
> > > We agree to make it pluggable. There're two allocation algorithms:
> > > server-session mapping and consistent hashing. We haven't agreed on
> which
> > > one is the default.
> > > 2. The semantic change of getAllSessions
> > > We agree to not change its semantics. However, we haven't achieved
> > > an agreement on the implementation.
> > >
> > > We think the open questions are not a blocker. So can we have this
> design
> > > doc approved?
> > >
> > > Thanks
> > > Yiheng
> > >
> > > On Thu, Nov 28, 2019 at 6:37 PM Yiheng Wang  wrote:
> > >
> > > > Hi Community
> > > >
> > > > We propose a multi-active high availability design for Livy. We think
> > > this
> > > > feature is valuable for many users. Here're the JIRA and the design
> > doc.
> > > If
> > > > you're interested, please take a look and comment :-)
> > > >
> > > > JIRA:
> > > > https://issues.apache.org/jira/projects/LIVY/issues/LIVY-718
> > > >
> > > > Design doc:
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
> > > >
> > > > Best regards
> > > > Yiheng
> > > >
> > >
> >
>

Re: Support multi-active high availability in Livy

[Saisai Shao]

I think it is feasible to move to next step. Let's create subtasks to track
all the works.

Thanks
Saisai

Yiheng Wang  于2019年12月10日周二 下午3:18写道：

> Hi Community
>
> We have received some feedback for this design. Here're the open questions
> for discussion
> 1. Session allocation algorithm
> We agree to make it pluggable. There're two allocation algorithms:
> server-session mapping and consistent hashing. We haven't agreed on which
> one is the default.
> 2. The semantic change of getAllSessions
> We agree to not change its semantics. However, we haven't achieved
> an agreement on the implementation.
>
> We think the open questions are not a blocker. So can we have this design
> doc approved?
>
> Thanks
> Yiheng
>
> On Thu, Nov 28, 2019 at 6:37 PM Yiheng Wang  wrote:
>
> > Hi Community
> >
> > We propose a multi-active high availability design for Livy. We think
> this
> > feature is valuable for many users. Here're the JIRA and the design doc.
> If
> > you're interested, please take a look and comment :-)
> >
> > JIRA:
> > https://issues.apache.org/jira/projects/LIVY/issues/LIVY-718
> >
> > Design doc:
> >
> >
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
> >
> > Best regards
> > Yiheng
> >
>

Re: Plan to cut new branch and prepare to release 0.7

[Saisai Shao]

Hi all,

I've cut the branch-0.7
https://github.com/apache/incubator-livy/tree/branch-0.7 and prepare for
the release. Now master branch is bumped to 0.8.0-incubating-SNAPSHOT.

Best regards,
Saisai

yihengw  于2019年10月11日周五 上午10:31写道：

> Yes, the binary mode has this problem.
>
>
> In http mode, jdbc clients talk to thrift server not using a long
> connection. It should be fit the recovery.
>
>
>  原始邮件
> 发件人: Marco Gaido
> 收件人: dev
> 发送时间: 2019年10月10日(周四) 21:59
> 主题: Re: Plan to cut new branch and prepare to release 0.7
>
>
> I think it is harder to do that for the thriftserver case. By its nature,
> the REST API allows multiple single requests, while in the thrifserver case
> the connection is always the same. I think without automatic failover, that
> feature is not so critical. Il giorno gio 10 ott 2019 alle ore 11:12
> yihengw  ha scritto: > Regarding the thrift server GA,
> should we consider supporting recovery in > thrift server? I open a JIRA
> for it > https://issues.apache.org/jira/browse/LIVY-696 > > > welcome for
> discussion. > > > 原始邮件 > 发件人: Saisai Shao > 收件人:
> dev > 发送时间: 2019年10月10日(周四) 16:24 > 主题:
> Re: Plan to cut new branch and prepare to release 0.7 > > > I'm hesitant to
> include service discovery into 0.7.0. Because all of the > JIRAs only
> addressed a small piece of high availability, but lack a whole > picture of
> HA design, I would suggest to have a whole HA design, then > splitting into
> small pieces, rather than opposite. So currently I'm not > confident if the
> JIRAs are good enough to merge. Thanks Saisai Marco Gaido < >
> marcogaid...@gmail.com> 于2019年10月10日周四 下午4:19写道： > Hi Saisai, > > thanks
> > for starting this discussion. I agree starting talking about a 0.7.0 > >
> release. Just a couple of notes: > > [LIVY-692][THRIFT] Use configured >
> principal for Kerberos authentication: as > I mentioned in the PR, I don't
> > think this is an actual problem. > > What about including in the scope of
> > this release also service discovery? > I've seen a couple of PRs and it
> may > be worth to have it in this release. > Any thoughts on this? > >
> Thanks, > > Marco > > Il giorno gio 10 ott 2019 alle ore 06:08 Jeff Zhang <
> > zjf...@gmail.com> ha > scritto: > > > Make sense to me > > > > Saisai >
> Shao  于2019年10月10日周四 下午12:04写道： > > > > > Hi all,
> > > > > > > > I'm planning to cut a new branch and prepare for the 0.7 >
> release, which > > did > > > lots of improvements to thrift module, I think
> > we could make thrift > > module > > > GA as for this release. > > > > > >
> > Currently I'm waiting for this JIRAs to be merged before cutting the >
> new > > > > branch. > > > > > > [LIVY-356][SERVER]Add LDAP authentication
> for > livy-server. > > > [LIVY-678] Thrift ldap authentication, based on
> ldapurl, > basedn, domain > > > [LIVY-692][THRIFT] Use configured principal
> for > Kerberos authentication > > > [LIVY-689] Deliver stage process
> message to > the end user using > > thriftserver > > > > > > Any thought? >
> > > > > > > Thanks > > > Saisai > > > > > > > > > -- > > Best Regards > > >
> > Jeff > Zhang > > >

Re: Enabling fully disaggregated shuffle on Spark

[Saisai Shao]

Hi Ben and Felix, I'm also interested in this. Would you please add me to
the invite, thanks a lot.

Best regards,
Saisai

Greg Lee  于2019年12月2日周一 下午11:34写道：

> Hi Felix & Ben,
>
> This is Li Hao from Baidu, same team with Linhong.
>
> As mentioned in Linhong’s email, independent disaggregated shuffle service
> is also our solution and continuous exploring direction for  improving
> stability of Hadoop MR and Spark in the production environment. We would
> love to hear about this topic from community and share our experience .
>
> Please add me to this event, thanks.
>
> Best Regards
> Li Hao
>
> Liu,Linhong  于2019年11月29日周五 下午5:09写道：
>
>> Hi Felix & Ben,
>>
>> This is Linhong from Baidu based in Beijing, and we are internally using
>> a disaggregated shuffle service (we call it DCE) as well. We launched this
>> in production 3 years ago for Hadoop shuffle. Last year we migrated spark
>> shuffle to the same DCE shuffle service and stability improved a lot (we
>> can handle more than 100T shuffle now).
>>
>> It would be nice if there is a Spark shuffle API support fully
>> disaggregated shuffle and my team and I are very glad to share our
>> experience and help on this topic.
>>
>> So, if It’s possible, please add me to this event.
>>
>>
>>
>> Thanks,
>>
>> Liu, Linhong
>>
>>
>>
>> *From: *Aniket Mokashi 
>> *Date: *Thursday, November 21, 2019 at 2:12 PM
>> *To: *Felix Cheung 
>> *Cc: *Ben Sidhom , John Zhuge <
>> jzh...@apache.org>, bo yang , Amogh Margoor <
>> amo...@qubole.com>, Ryan Blue , Spark Dev List <
>> dev@spark.apache.org>, Christopher Crosbie ,
>> Griselda Cuevas , Holden Karau ,
>> Mayank Ahuja , Kalyan Sivakumar ,
>> "alfo...@fb.com" , Felix Cheung , Matt
>> Cheah , "Yifei Huang (PD)" 
>> *Subject: *Re: Enabling fully disaggregated shuffle on Spark
>>
>>
>>
>> Felix - please add me to this event.
>>
>>
>>
>> Ben - should we move this proposal to a doc and open it up for
>> edits/comments.
>>
>>
>>
>> On Wed, Nov 20, 2019 at 5:37 PM Felix Cheung 
>> wrote:
>>
>> Great!
>>
>>
>>
>> Due to number of constraints I won’t be sending link directly here but
>> please r me and I will add you.
>>
>>
>>
>>
>> --
>>
>> *From:* Ben Sidhom 
>> *Sent:* Wednesday, November 20, 2019 9:10:01 AM
>> *To:* John Zhuge 
>> *Cc:* bo yang ; Amogh Margoor ;
>> Ryan Blue ; Ben Sidhom ;
>> Spark Dev List ; Christopher Crosbie <
>> crosb...@google.com>; Griselda Cuevas ; Holden Karau <
>> hol...@pigscanfly.ca>; Mayank Ahuja ; Kalyan
>> Sivakumar ; alfo...@fb.com ; Felix
>> Cheung ; Matt Cheah ; Yifei Huang
>> (PD) 
>> *Subject:* Re: Enabling fully disaggregated shuffle on Spark
>>
>>
>>
>> That sounds great!
>>
>>
>>
>> On Wed, Nov 20, 2019 at 9:02 AM John Zhuge  wrote:
>>
>> That will be great. Please send us the invite.
>>
>>
>>
>> On Wed, Nov 20, 2019 at 8:56 AM bo yang  wrote:
>>
>> Cool, thanks Ryan, John, Amogh for the reply! Great to see you
>> interested! Felix will have a Spark Scalability & Reliability Sync
>> meeting on Dec 4 1pm PST. We could discuss more details there. Do you want
>> to join?
>>
>>
>>
>> On Tue, Nov 19, 2019 at 4:23 PM Amogh Margoor  wrote:
>>
>> We at Qubole are also looking at disaggregating shuffle on Spark. Would
>> love to collaborate and share learnings.
>>
>>
>>
>> Regards,
>>
>> Amogh
>>
>>
>>
>> On Tue, Nov 19, 2019 at 4:09 PM John Zhuge  wrote:
>>
>> Great work, Bo! Would love to hear the details.
>>
>>
>>
>>
>>
>> On Tue, Nov 19, 2019 at 4:05 PM Ryan Blue 
>> wrote:
>>
>> I'm interested in remote shuffle services as well. I'd love to hear about
>> what you're using in production!
>>
>>
>>
>> rb
>>
>>
>>
>> On Tue, Nov 19, 2019 at 2:43 PM bo yang  wrote:
>>
>> Hi Ben,
>>
>>
>>
>> Thanks for the writing up! This is Bo from Uber. I am in Felix's team in
>> Seattle, and working on disaggregated shuffle (we called it remote shuffle
>> service, RSS, internally). We have put RSS into production for a while, and
>> learned a lot during the work (tried quite a few techniques to improve the
>> remote shuffle performance). We could share our learning with the
>> community, and also would like to hear feedback/suggestions on how to
>> further improve remote shuffle performance. We could chat more details if
>> you or other people are interested.
>>
>>
>>
>> Best,
>>
>> Bo
>>
>>
>>
>> On Fri, Nov 15, 2019 at 4:10 PM Ben Sidhom 
>> wrote:
>>
>> I would like to start a conversation about extending the Spark shuffle
>> manager surface to support fully disaggregated shuffle implementations.
>> This is closely related to the work in SPARK-25299
>> , which is focused on
>> refactoring the shuffle manager API (and in particular, SortShuffleManager)
>> to use a pluggable storage backend. The motivation for that SPIP is further
>> enabling Spark on Kubernetes.
>>
>>
>>
>> The motivation for this proposal is enabling full externalized
>> (disaggregated) shuffle service implementations. (Facebook’s Cosco
>> shuffle
>> 

[jira] [Resolved] (LIVY-717) Specify explicit ZooKeeper version in maven

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-717.
--
Fix Version/s: 0.7.0
 Assignee: Mate Szalay-Beko
   Resolution: Fixed

Issue resolved by pull request 262
https://github.com/apache/incubator-livy/pull/262

> Specify explicit ZooKeeper version in maven
> ---
>
> Key: LIVY-717
> URL: https://issues.apache.org/jira/browse/LIVY-717
> Project: Livy
>  Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Hadoop trunk was updated recently to use Curator 4.0.2 and ZooKeeper 3.5.6, 
> see [HADOOP-16579|https://issues.apache.org/jira/browse/HADOOP-16579]. Now I 
> want to test Livy in a cluster where we have a new ZooKeeper 3.5 deployed. 
> When we want to use Livy in a cluster where a newer ZooKeeper server version 
> is used, we might run into run-time errors if we compile Livy using the 
> current Curator / Hadoop versions. The Curator version can already explicitly 
> set with the {{curator.version}} maven property in build time, but we were 
> still missed the same parameter for ZooKeeper.
> In this PR I added a new maven parameter called {{zookeeper.version}} and 
> after analyzing the maven dependency tree, I made sure that the Curator and 
> ZooKeeper versions used in compile time are always harmonized and controlled 
> by the maven parameters.
> I set the zookeeper.version in maven to {{3.4.6}} to be backward compatible 
> with the current Livy dependencies.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-714) Cannot remove the app in leakedAppTags when timeout

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-714.
--
Fix Version/s: 0.7.0
 Assignee: runzhiwang
   Resolution: Fixed

Issue resolved by pull request 259
https://github.com/apache/incubator-livy/pull/259

> Cannot remove the app in leakedAppTags when timeout
> ---
>
> Key: LIVY-714
> URL: https://issues.apache.org/jira/browse/LIVY-714
> Project: Livy
>  Issue Type: New Feature
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: image-2019-11-20-09-50-52-316.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> # var isRemoved = false should be in while(iter.hasNext), otherwise if there 
> are two apps, the first app will be removed and the second app will timeout 
> in this loop, and after remove the first app, isRemoved = true, and the 
> second app cannot pass the if(!isRemoved) and only will be delete in the next 
> loop.
>  # entry.getValue - now is negative, and never greater than 
> sessionLeakageCheckTimeout.
> !image-2019-11-20-09-50-52-316.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Iceberg in Spark 3.0.0

[Saisai Shao]

Thanks guys for your reply.

Hi Ryan, would you please help to create a spark-3.0 branch, so we could
submit our PRs against that branch.

Best regards,
Saisai

Ryan Blue  于2019年11月23日周六 上午2:03写道：

> I agree, let's create a spark-3.0 branch to start with. We've been
> building vectorization this way using the vectorized-reads branch.
>
> In the long term, we may want to split Spark into separate modules for 2.x
> and 3.x in the same branch, but for now we can at least get everything
> working with a 3.0 branch.
>
> On Fri, Nov 22, 2019 at 8:34 AM John Zhuge  wrote:
>
>> +1 for Iceberg branch
>>
>> Thanks for the contribution from you and your team!
>>
>> On Fri, Nov 22, 2019 at 8:29 AM Anton Okolnychyi
>>  wrote:
>>
>>> +1 on having a branch in Iceberg as we have for vectorized reads.
>>>
>>> - Anton
>>>
>>> On 22 Nov 2019, at 02:26, Saisai Shao  wrote:
>>>
>>> Hi Ryan and team,
>>>
>>> Thanks a lot for your response. I was wondering how do we share our
>>> branch, one possible way s that we maintain a forked Iceberg repo with
>>> Spark 3.0.0-preview branch, another possible way is to create a branch in
>>> upstream Iceberg repo. I'm inclined to choose the second way, so that they
>>> community could review and contribute on it.
>>>
>>> I would like to hear your suggestions.
>>>
>>> Best regards,
>>> Saisai
>>>
>>>
>>> Ryan Blue  于2019年11月20日周三 上午1:27写道：
>>>
>>>> Sounds great, thanks Saisai!
>>>>
>>>> On Mon, Nov 18, 2019 at 3:29 AM Saisai Shao 
>>>> wrote:
>>>>
>>>>> Thanks Anton, I will share our branch soon.
>>>>>
>>>>> Best regards,
>>>>> Saisai
>>>>>
>>>>> Anton Okolnychyi  于2019年11月18日周一
>>>>> 下午6:54写道：
>>>>>
>>>>>> I think it would be great if you can share what you have, Saisai.
>>>>>> That way, we can all collaborate and ensure we build a full 3.0 
>>>>>> integration
>>>>>> as soon as possible.
>>>>>>
>>>>>> - Anton
>>>>>>
>>>>>>
>>>>>> On 18 Nov 2019, at 02:08, Saisai Shao  wrote:
>>>>>>
>>>>>> Hi Anton,
>>>>>>
>>>>>> Thanks to bring this out. We already have a branch building against
>>>>>> Spark 3.0 (Master branch actually) internally, and we're actively working
>>>>>> on it. I think it is a good idea to create an upstream Spark 3.0 branch, 
>>>>>> we
>>>>>> could share it if the community would like to do so.
>>>>>>
>>>>>> Best regards,
>>>>>> Saisai
>>>>>>
>>>>>> Anton Okolnychyi  于2019年11月18日周一
>>>>>> 上午1:40写道：
>>>>>>
>>>>>>> I think it is a good time to create a branch to build our 3.0
>>>>>>> integration as the 3.0 preview was released.
>>>>>>> What does everyone think? Has anybody started already?
>>>>>>>
>>>>>>> - Anton
>>>>>>>
>>>>>>> On 8 Aug 2019, at 23:47, Edgar Rodriguez 
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 8, 2019 at 3:37 PM Ryan Blue  wrote:
>>>>>>>
>>>>>>>> I think it's a great idea to branch and get ready for Spark 3.0.0.
>>>>>>>> Right now, I'm focused on getting a release out, but I can review 
>>>>>>>> patches
>>>>>>>> for Spark 3.0.
>>>>>>>>
>>>>>>>> Anyone know if there are nightly builds of Spark 3.0 to test with?
>>>>>>>>
>>>>>>>
>>>>>>> Seems like there're nightly snapshots built in
>>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-sql_2.12/3.0.0-SNAPSHOT/
>>>>>>>  -
>>>>>>> I've started setting something up with these snapshots so I can probably
>>>>>>> start working on this.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Cheers,
>>>>>>> --
>>>>>>> Edgar Rodriguez
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
>>>
>>
>> --
>> John Zhuge
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Query about the semantics of "overwrite" in Iceberg

[Saisai Shao]

Thanks Ryan for your response, let me spend more on Spark 3.0 overwrite
behavior.

Best Regards
Saisai

Ryan Blue  于2019年11月23日周六 上午1:08写道：

> Saisai,
>
> Iceberg's behavior matches Hive's and Spark's behavior when using dynamic
> overwrite mode.
>
> Spark does not specify the correct behavior -- it varies by source. In
> addition, it isn't possible for a v2 source in 2.4 to implement the static
> overwrite mode that is Spark's default. The problem is that the source is
> not passed the static partition values, only rows.
>
> This is fixed in 3.0 because Spark will choose its behavior and correctly
> configure the source with a dynamic overwrite or an overwrite using an
> expression.
>
> On Thu, Nov 21, 2019 at 11:33 PM Saisai Shao 
> wrote:
>
>> Hi Team,
>>
>> I found that Iceberg's "overwrite" is different from Spark's built-in
>> sources like Parquet. The "overwrite" semantics in Iceberg seems more like
>> "upsert", but not deleting the partitions where new data doesn't contain.
>>
>> I would like to know what is the purpose of such design choice? Also if I
>> want to achieve Spark Parquet's "overwrite" semantics, how would I
>> achieve this?
>>
>> Warning
>>
>> *Spark does not define the behavior of DataFrame overwrite*. Like most
>> sources, Iceberg will dynamically overwrite partitions when the dataframe
>> contains rows in a partition. Unpartitioned tables are completely
>> overwritten.
>>
>> Best regards,
>> Saisai
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Query about the semantics of "overwrite" in Iceberg

[Saisai Shao]

Hi Team,

I found that Iceberg's "overwrite" is different from Spark's built-in
sources like Parquet. The "overwrite" semantics in Iceberg seems more like
"upsert", but not deleting the partitions where new data doesn't contain.

I would like to know what is the purpose of such design choice? Also if I
want to achieve Spark Parquet's "overwrite" semantics, how would I
achieve this?

Warning

*Spark does not define the behavior of DataFrame overwrite*. Like most
sources, Iceberg will dynamically overwrite partitions when the dataframe
contains rows in a partition. Unpartitioned tables are completely
overwritten.

Best regards,
Saisai

[jira] [Resolved] (LIVY-715) The configuration in the livy.conf.template is inconsistent with LivyConf.scala

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-715.
--
Fix Version/s: 0.7.0
   Resolution: Fixed

Issue resolved by pull request 261
https://github.com/apache/incubator-livy/pull/261

> The configuration in the livy.conf.template is inconsistent with 
> LivyConf.scala
> ---
>
> Key: LIVY-715
> URL: https://issues.apache.org/jira/browse/LIVY-715
> Project: Livy
>  Issue Type: Bug
>  Components: Docs
>Affects Versions: 0.6.0
>Reporter: mingchao zhao
>Assignee: mingchao zhao
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
>     When I test livy impersonation found that, in livy.conf.template the 
> value of livy.impersonation.enabled is true. So I thought impersonation was 
> enabled by default.             
>      However, impersonation was not turned on when we test. I found that the 
> real configuration in LivyConf. scala is false. This can mislead users.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Iceberg in Spark 3.0.0

[Saisai Shao]

Hi Ryan and team,

Thanks a lot for your response. I was wondering how do we share our branch,
one possible way s that we maintain a forked Iceberg repo with Spark
3.0.0-preview branch, another possible way is to create a branch in
upstream Iceberg repo. I'm inclined to choose the second way, so that they
community could review and contribute on it.

I would like to hear your suggestions.

Best regards,
Saisai


Ryan Blue  于2019年11月20日周三 上午1:27写道：

> Sounds great, thanks Saisai!
>
> On Mon, Nov 18, 2019 at 3:29 AM Saisai Shao 
> wrote:
>
>> Thanks Anton, I will share our branch soon.
>>
>> Best regards,
>> Saisai
>>
>> Anton Okolnychyi  于2019年11月18日周一 下午6:54写道：
>>
>>> I think it would be great if you can share what you have, Saisai. That
>>> way, we can all collaborate and ensure we build a full 3.0 integration as
>>> soon as possible.
>>>
>>> - Anton
>>>
>>>
>>> On 18 Nov 2019, at 02:08, Saisai Shao  wrote:
>>>
>>> Hi Anton,
>>>
>>> Thanks to bring this out. We already have a branch building against
>>> Spark 3.0 (Master branch actually) internally, and we're actively working
>>> on it. I think it is a good idea to create an upstream Spark 3.0 branch, we
>>> could share it if the community would like to do so.
>>>
>>> Best regards,
>>> Saisai
>>>
>>> Anton Okolnychyi  于2019年11月18日周一
>>> 上午1:40写道：
>>>
>>>> I think it is a good time to create a branch to build our 3.0
>>>> integration as the 3.0 preview was released.
>>>> What does everyone think? Has anybody started already?
>>>>
>>>> - Anton
>>>>
>>>> On 8 Aug 2019, at 23:47, Edgar Rodriguez 
>>>> wrote:
>>>>
>>>>
>>>>
>>>> On Thu, Aug 8, 2019 at 3:37 PM Ryan Blue  wrote:
>>>>
>>>>> I think it's a great idea to branch and get ready for Spark 3.0.0.
>>>>> Right now, I'm focused on getting a release out, but I can review patches
>>>>> for Spark 3.0.
>>>>>
>>>>> Anyone know if there are nightly builds of Spark 3.0 to test with?
>>>>>
>>>>
>>>> Seems like there're nightly snapshots built in
>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-sql_2.12/3.0.0-SNAPSHOT/
>>>>  -
>>>> I've started setting something up with these snapshots so I can probably
>>>> start working on this.
>>>>
>>>> Thanks!
>>>>
>>>> Cheers,
>>>> --
>>>> Edgar Rodriguez
>>>>
>>>>
>>>>
>>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Iceberg in Spark 3.0.0

[Saisai Shao]

Thanks Anton, I will share our branch soon.

Best regards,
Saisai

Anton Okolnychyi  于2019年11月18日周一 下午6:54写道：

> I think it would be great if you can share what you have, Saisai. That
> way, we can all collaborate and ensure we build a full 3.0 integration as
> soon as possible.
>
> - Anton
>
>
> On 18 Nov 2019, at 02:08, Saisai Shao  wrote:
>
> Hi Anton,
>
> Thanks to bring this out. We already have a branch building against Spark
> 3.0 (Master branch actually) internally, and we're actively working on it.
> I think it is a good idea to create an upstream Spark 3.0 branch, we could
> share it if the community would like to do so.
>
> Best regards,
> Saisai
>
> Anton Okolnychyi  于2019年11月18日周一 上午1:40写道：
>
>> I think it is a good time to create a branch to build our 3.0 integration
>> as the 3.0 preview was released.
>> What does everyone think? Has anybody started already?
>>
>> - Anton
>>
>> On 8 Aug 2019, at 23:47, Edgar Rodriguez 
>> wrote:
>>
>>
>>
>> On Thu, Aug 8, 2019 at 3:37 PM Ryan Blue  wrote:
>>
>>> I think it's a great idea to branch and get ready for Spark 3.0.0. Right
>>> now, I'm focused on getting a release out, but I can review patches for
>>> Spark 3.0.
>>>
>>> Anyone know if there are nightly builds of Spark 3.0 to test with?
>>>
>>
>> Seems like there're nightly snapshots built in
>> https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-sql_2.12/3.0.0-SNAPSHOT/
>>  -
>> I've started setting something up with these snapshots so I can probably
>> start working on this.
>>
>> Thanks!
>>
>> Cheers,
>> --
>> Edgar Rodriguez
>>
>>
>>
>

Re: Iceberg in Spark 3.0.0

[Saisai Shao]

Hi Anton,

Thanks to bring this out. We already have a branch building against Spark
3.0 (Master branch actually) internally, and we're actively working on it.
I think it is a good idea to create an upstream Spark 3.0 branch, we could
share it if the community would like to do so.

Best regards,
Saisai

Anton Okolnychyi  于2019年11月18日周一 上午1:40写道：

> I think it is a good time to create a branch to build our 3.0 integration
> as the 3.0 preview was released.
> What does everyone think? Has anybody started already?
>
> - Anton
>
> On 8 Aug 2019, at 23:47, Edgar Rodriguez 
> wrote:
>
>
>
> On Thu, Aug 8, 2019 at 3:37 PM Ryan Blue  wrote:
>
>> I think it's a great idea to branch and get ready for Spark 3.0.0. Right
>> now, I'm focused on getting a release out, but I can review patches for
>> Spark 3.0.
>>
>> Anyone know if there are nightly builds of Spark 3.0 to test with?
>>
>
> Seems like there're nightly snapshots built in
> https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-sql_2.12/3.0.0-SNAPSHOT/
>  -
> I've started setting something up with these snapshots so I can probably
> start working on this.
>
> Thanks!
>
> Cheers,
> --
> Edgar Rodriguez
>
>
>

[jira] [Resolved] (LIVY-707) Add audit log for SqlJobs from ThriftServer

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-707.
--
Fix Version/s: 0.7.0
 Assignee: Hui An
   Resolution: Fixed

Issue resolved by pull request 255
https://github.com/apache/incubator-livy/pull/255

> Add audit log for SqlJobs from ThriftServer
> ---
>
> Key: LIVY-707
> URL: https://issues.apache.org/jira/browse/LIVY-707
> Project: Livy
>  Issue Type: Improvement
>  Components: Thriftserver
>Reporter: Hui An
>Assignee: Hui An
>Priority: Minor
> Fix For: 0.7.0
>
>
> The audit Log style is below
> {code:java}
> 19/11/06 16:38:30 INFO ThriftServerAudit$: user: test ipAddress: 10.25.22.46 
> query: select count(*) from test1 beforeExecute: 1573029416951 afterExecute: 
> 1573029510972 time spent: 94021
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (LIVY-711) Travis fails to build on Ubuntu16.04

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated LIVY-711:
-
Component/s: Build

> Travis fails to build on Ubuntu16.04
> 
>
> Key: LIVY-711
> URL: https://issues.apache.org/jira/browse/LIVY-711
> Project: Livy
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 0.6.0
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: image-2019-11-12-10-16-27-108.png, 
> image-2019-11-12-14-25-37-189.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> !image-2019-11-12-14-25-37-189.png!!image-2019-11-12-10-16-27-108.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (LIVY-711) Travis fails to build on Ubuntu16.04

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated LIVY-711:
-
Affects Version/s: 0.6.0

> Travis fails to build on Ubuntu16.04
> 
>
> Key: LIVY-711
> URL: https://issues.apache.org/jira/browse/LIVY-711
> Project: Livy
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: image-2019-11-12-10-16-27-108.png, 
> image-2019-11-12-14-25-37-189.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> !image-2019-11-12-14-25-37-189.png!!image-2019-11-12-10-16-27-108.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (LIVY-711) Travis fails to build on Ubuntu16.04

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated LIVY-711:
-
Issue Type: Bug  (was: New Feature)

> Travis fails to build on Ubuntu16.04
> 
>
> Key: LIVY-711
> URL: https://issues.apache.org/jira/browse/LIVY-711
> Project: Livy
>  Issue Type: Bug
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: image-2019-11-12-10-16-27-108.png, 
> image-2019-11-12-14-25-37-189.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> !image-2019-11-12-14-25-37-189.png!!image-2019-11-12-10-16-27-108.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-711) Travis fails to build on Ubuntu16.04

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-711.
--
Fix Version/s: 0.7.0
 Assignee: runzhiwang
   Resolution: Fixed

Issue resolved by pull request 257
https://github.com/apache/incubator-livy/pull/257

> Travis fails to build on Ubuntu16.04
> 
>
> Key: LIVY-711
> URL: https://issues.apache.org/jira/browse/LIVY-711
> Project: Livy
>  Issue Type: New Feature
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: image-2019-11-12-10-16-27-108.png, 
> image-2019-11-12-14-25-37-189.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> !image-2019-11-12-14-25-37-189.png!!image-2019-11-12-10-16-27-108.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-708) The version of curator jars are not aligned

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-708.
--
Fix Version/s: 0.7.0
 Assignee: Yiheng Wang
   Resolution: Fixed

Issue resolved by pull request 256
https://github.com/apache/incubator-livy/pull/256

> The version of curator jars are not aligned
> ---
>
> Key: LIVY-708
> URL: https://issues.apache.org/jira/browse/LIVY-708
> Project: Livy
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 0.6.0
>Reporter: Yiheng Wang
>Assignee: Yiheng Wang
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Livy server has dependency of Apache Curator through hadoop client. However, 
> the versions of the curator jars are not aligned. Here're the curator jars 
> after build
> * curator-client-2.7.1.jar
> * curator-framework-2.7.1.jar
> * curator-recipes-2.6.0.jar
> This will cause Method not found issue in some case:
> Exception in thread "main" java.lang.NoSuchMethodError: 
> {code:bash}
> org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)V
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: livy and security concerns

[Saisai Shao]

Livy is just a gateway service, I think it is not Livy's responsibility to
provide a sandbox. Also it is super hard to analyze the code from Livy
side. Typically some companies will provide a sand-boxing runtime, like JVM
or others, or put Spark into container for security and isolation.

Thanks
Saisai

Meisam Fathi  于2019年11月1日周五 上午2:57写道：

> No, it does not. The code will run on a Spark session. The sandboxing
> should be supported, at least partially, by Spark. But as far as I know,
> Spark dies not have such a feature.
>
> Thanks,
> Meisam
>
> On Wed, Oct 30, 2019, 5:02 PM mhd wrk  wrote:
>
>> Considering that Livy supports Interactive Scala or Python, does it
>> provides any sand-boxing feature to protect the back-end against submitted
>> code?
>>
>

[jira] [Resolved] (LIVY-678) Livy Thrift-server Ordinary ldap authentication, based on ldap.url, basedn, domain

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-678.
--
Fix Version/s: 0.7.0
 Assignee: mingchao zhao
   Resolution: Fixed

Issue resolved by pull request 236
https://github.com/apache/incubator-livy/pull/236

> Livy Thrift-server Ordinary ldap authentication, based on ldap.url, basedn, 
> domain
> --
>
> Key: LIVY-678
> URL: https://issues.apache.org/jira/browse/LIVY-678
> Project: Livy
>  Issue Type: Sub-task
>  Components: Thriftserver
>Reporter: mingchao zhao
>Assignee: mingchao zhao
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Add provider  to search for the user in the LDAP directory within the baseDN 
> tree. Access is granted if user has been found, and denied otherwise.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Accept TubeMQ into the Apache Incubator

[Saisai Shao]

+1 (non-binding).

Thanks
Saisai

Sheng Wu  于2019年10月30日周三 上午10:03写道：

> +1 binding
> Good luck.
>
> Dave Fisher 于2019年10月30日 周三上午7:30写道：
>
> > +1 (binding)
> >
> > Go TubeMQ.
> >
> > Regards,
> > Dave
> >
> > Sent from my iPhone
> >
> > > On Oct 29, 2019, at 6:48 PM, David Nalley  wrote:
> > >
> > > Hi folks,
> > >
> > > The [DISCUSS] thread on TubeMQ has died down.
> > >
> > > Accordingly, I would like to call a VOTE to accept TubeMQ into the
> > > Apache Incubator.
> > >
> > > Please cast your vote:
> > >
> > >  [ ] +1, bring TubeMQ into the Incubator
> > >  [ ] +0, I don't care either way
> > >  [ ] -1, do not bring TubeMQ into the Incubator, because...
> > >
> > > The vote will open at least for 72 hours and only votes from the
> > > Incubator PMC are binding, but votes from everyone are welcome.
> > >
> > > =Abstract=
> > >
> > > TubeMQ is a distributed messaging queue (MQ) system developed by
> > > Tencent Big Data since 2013. It focuses on high-performance storage
> > > and transmission of massive data in big data scenarios.After nearly
> > > seven years of massive data precipitation, TubeMQ has certain
> > > advantages in production practice (stability + performance) and low
> > > cost compared to many open source MQ projects.
> > >
> > > =Proposal=
> > >
> > > TubeMQ is suitable for high concurrency, massive data and tolerates a
> > > small amount of data loss scenarios under abnormal conditions, such as
> > > massive log collection, indicator statistics and monitoring, etc.
> > > TubeMQ does not support highly reliable data transmission services
> > > yet. It could be on a future project roadmap, as many other MQs. but
> > > not today.
> > >
> > > =Rationale=
> > >
> > > Just like other message queue systems, TubeMQ is built on the
> > > publish-subscribe pattern, aka pub-sub.
> > > In this pattern, producers publish messages to topics while consumers
> > > subscribe to those topics. After incoming messages get proceeded,
> > > consumers send an acknowledgement back to producer. Once a
> > > subscription has been created, all messages will be tracked and
> > > retained by TubeMQ, even if the consumer go offline for some reasons.
> > > Retained messages will be discarded only when a consumer acknowledges
> > > that they've been successfully processed.
> > >
> > > Portal is responsible for interact with user and admin system which
> > > include two parts: API and web portal.
> > >
> > > Master is controller of the cluster, which include one or multiple
> > > master node(s) which is responsible for managing state of cluster,
> > > resource scheduling, authentication check and maintaining of metadata.
> > > As a reliable system, TubeMQ provides HA solution for master node.
> > >
> > > Broker is responsible for data store which include a cluster of broker
> > > nodes. Every broker node is managing a set of topics, include: append,
> > > delete, update, query of topic information. In TubeMQ, these brokers
> > > can be horizontal scaled and can be very large size for massive data
> > > case.
> > >
> > > Client is responsible for producing and consuming messages. When a
> > > pub-sub topic get setup, we can support two ways (push and pull) for
> > > delivering message from producers to consumers.
> > >
> > > Zookeeper is for storing offset of messages which is used to recover
> > > topic during some components get failed.
> > >
> > >
> > > =Initial Goals=
> > >
> > > The initial goal will be to move the current codebase in github’s
> > > repository under Tencent account to Apache and integrate with the
> > > Apache development process and infrastructure.
> > > A primary goal of incubation will be to grow and diversify the TubeMQ
> > > community. We are well aware that the project community is largely
> > > comprised of individuals from a single company. We aim to change that
> > > during incubation.
> > >
> > > =Current Status=
> > >
> > > As previously mentioned, TubeMQ is under active development at
> > > Tencent, and is being used in processing large volumes of data for
> > > most services and products.
> > >
> > > =Meritocracy=
> > >
> > > We value meritocracy and we understand that it is the basis for an
> > > open community that encourages multiple companies and individuals to
> > > contribute and be invested in the project’s future. We will encourage
> > > and monitor participation and make sure to extend privileges and
> > > responsibilities to all contributors.
> > >
> > > =Community=
> > >
> > > TubeMQ is currently being used by developers at Tencent and a growing
> > > number of users are actively using it in production environments.
> > > TubeMQ has received contributions from developers working outside of
> > > Tencent since it was open sourced on github in September 2019 By
> > > bringing TubeMQ to Apache we aim to assure current and future
> > > contributors that the TubeMQ community is neutral, meritocratic, and
> > > open, in order to broaden and diversity the user and developer
> > > 

[jira] [Resolved] (LIVY-697) Rsc client cannot resolve the hostname of driver in yarn-cluster mode

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-697.
--
Fix Version/s: 0.7.0
 Assignee: runzhiwang
   Resolution: Fixed

Issue resolved by pull request 246
https://github.com/apache/incubator-livy/pull/246

> Rsc client cannot resolve the hostname of driver in yarn-cluster mode
> -
>
> Key: LIVY-697
> URL: https://issues.apache.org/jira/browse/LIVY-697
> Project: Livy
>  Issue Type: Bug
>  Components: RSC
>Affects Versions: 0.6.0
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: image-2019-10-13-12-44-41-861.png
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> !image-2019-10-13-12-44-41-861.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Plan to cut new branch and prepare to release 0.7

[Saisai Shao]

I'm hesitant to include service discovery into 0.7.0. Because all of the
JIRAs only addressed a small piece of high availability, but lack a whole
picture of HA design, I would suggest to have a whole HA design, then
splitting into small pieces, rather than opposite. So currently I'm not
confident if the JIRAs are good enough to merge.

Thanks
Saisai

Marco Gaido  于2019年10月10日周四 下午4:19写道：

> Hi Saisai,
>
> thanks for starting this discussion. I agree starting talking about a 0.7.0
> release. Just a couple of notes:
>
> [LIVY-692][THRIFT] Use configured principal for Kerberos authentication: as
> I mentioned in the PR, I don't think this is an actual problem.
>
> What about including in the scope of this release also service discovery?
> I've seen a couple of PRs and it may be worth to have it in this release.
> Any thoughts on this?
>
> Thanks,
> Marco
>
> Il giorno gio 10 ott 2019 alle ore 06:08 Jeff Zhang  ha
> scritto:
>
> > Make sense to me
> >
> > Saisai Shao  于2019年10月10日周四 下午12:04写道：
> >
> > > Hi all,
> > >
> > > I'm planning to cut a new branch and prepare for the 0.7 release, which
> > did
> > > lots of improvements to thrift module, I think we could make thrift
> > module
> > > GA as for this release.
> > >
> > > Currently I'm waiting for this JIRAs to be merged before cutting the
> new
> > > branch.
> > >
> > > [LIVY-356][SERVER]Add LDAP authentication for livy-server.
> > > [LIVY-678] Thrift ldap authentication, based on ldapurl, basedn, domain
> > > [LIVY-692][THRIFT] Use configured principal for Kerberos authentication
> > > [LIVY-689] Deliver stage process message to the end user using
> > thriftserver
> > >
> > > Any thought?
> > >
> > > Thanks
> > > Saisai
> > >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>

Plan to cut new branch and prepare to release 0.7

[Saisai Shao]

Hi all,

I'm planning to cut a new branch and prepare for the 0.7 release, which did
lots of improvements to thrift module, I think we could make thrift module
GA as for this release.

Currently I'm waiting for this JIRAs to be merged before cutting the new
branch.

[LIVY-356][SERVER]Add LDAP authentication for livy-server.
[LIVY-678] Thrift ldap authentication, based on ldapurl, basedn, domain
[LIVY-692][THRIFT] Use configured principal for Kerberos authentication
[LIVY-689] Deliver stage process message to the end user using thriftserver

Any thought?

Thanks
Saisai

[jira] [Assigned] (LIVY-690) Exclude curator in thrift server pom to avoid conflict jars

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao reassigned LIVY-690:


Assignee: Yiheng Wang

> Exclude curator in thrift server pom to avoid conflict jars
> ---
>
> Key: LIVY-690
> URL: https://issues.apache.org/jira/browse/LIVY-690
> Project: Livy
>  Issue Type: Bug
>  Components: Thriftserver
>Affects Versions: 0.6.0
>Reporter: Yiheng Wang
>Assignee: Yiheng Wang
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently, thrift server has a dependency of curator-client:2.12.0 through 
> the hive service. After the build, a curator-client-2.12.0.jar file will be 
> generated in the jars folder. It is conflicted with the 
> curator-client-2.7.1.jar file, which is used by livy server.
> We observed that in some JDK, the curator-client-2.12.0.jar is loaded before 
> the curator-client-2.7.1.jar, and will crash the recovery enabled livy server.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-690) Exclude curator in thrift server pom to avoid conflict jars

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-690.
--
Resolution: Fixed

Issue resolved by pull request 239
[https://github.com/apache/incubator-livy/pull/239]

> Exclude curator in thrift server pom to avoid conflict jars
> ---
>
> Key: LIVY-690
> URL: https://issues.apache.org/jira/browse/LIVY-690
> Project: Livy
>  Issue Type: Bug
>  Components: Thriftserver
>Affects Versions: 0.6.0
>Reporter: Yiheng Wang
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently, thrift server has a dependency of curator-client:2.12.0 through 
> the hive service. After the build, a curator-client-2.12.0.jar file will be 
> generated in the jars folder. It is conflicted with the 
> curator-client-2.7.1.jar file, which is used by livy server.
> We observed that in some JDK, the curator-client-2.12.0.jar is loaded before 
> the curator-client-2.7.1.jar, and will crash the recovery enabled livy server.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (LIVY-688) Error message of LivyClient only keep outer stackTrace but discard cause's stackTrace

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao reassigned LIVY-688:


Assignee: weiwenda

> Error message of LivyClient only keep outer stackTrace but discard cause's 
> stackTrace
> -
>
> Key: LIVY-688
> URL: https://issues.apache.org/jira/browse/LIVY-688
> Project: Livy
>  Issue Type: Improvement
>  Components: RSC
>Affects Versions: 0.6.0
>Reporter: weiwenda
>Assignee: weiwenda
>Priority: Trivial
> Fix For: 0.7.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> 1. SparkSession maybe failed when initialize ExternalCatalog at sometime. As 
> SparkSession call _SharedState.reflect_ to instance an ExternalCatalog, any 
> exception happened during this process will wrapped by 
> InvocationTargetException.
> IllegalArgumentException
>    └──InvocationTargetException
>               └──the indeed Exception
>  2. org.apache.livy.rsc.Utils.stackTraceAsString only keep 
> IllegalArgumentException's stackTrace but discard the indeed Exception's 
> stackTrace and message, which makes the final 
> java.util.concurrent.ExecutionException's message ambiguous.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-688) Error message of LivyClient only keep outer stackTrace but discard cause's stackTrace

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-688.
--
Fix Version/s: 0.7.0
   Resolution: Fixed

Issue resolved by pull request 237
[https://github.com/apache/incubator-livy/pull/237]

> Error message of LivyClient only keep outer stackTrace but discard cause's 
> stackTrace
> -
>
> Key: LIVY-688
> URL: https://issues.apache.org/jira/browse/LIVY-688
> Project: Livy
>  Issue Type: Improvement
>  Components: RSC
>Affects Versions: 0.6.0
>Reporter: weiwenda
>Priority: Trivial
> Fix For: 0.7.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> 1. SparkSession maybe failed when initialize ExternalCatalog at sometime. As 
> SparkSession call _SharedState.reflect_ to instance an ExternalCatalog, any 
> exception happened during this process will wrapped by 
> InvocationTargetException.
> IllegalArgumentException
>    └──InvocationTargetException
>               └──the indeed Exception
>  2. org.apache.livy.rsc.Utils.stackTraceAsString only keep 
> IllegalArgumentException's stackTrace but discard the indeed Exception's 
> stackTrace and message, which makes the final 
> java.util.concurrent.ExecutionException's message ambiguous.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-658) RSCDriver should catch exception if cancel job failed during shutdown

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-658.
--
Fix Version/s: 0.7.0
 Assignee: Jeffrey(Xilang) Yan
   Resolution: Fixed

Issue resolved by pull request 223
https://github.com/apache/incubator-livy/pull/223

> RSCDriver should catch exception if cancel job failed during shutdown
> -
>
> Key: LIVY-658
> URL: https://issues.apache.org/jira/browse/LIVY-658
> Project: Livy
>  Issue Type: Bug
>  Components: RSC
>Reporter: Jeffrey(Xilang) Yan
>Assignee: Jeffrey(Xilang) Yan
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, if startup meet exception, exception will trigger spark to 
> shutdown, then trigger cancel job, but cancel job will throw another 
> exception due to spark is not initialized. The new exception will swallow the 
> old exception.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-644) Flaky test: Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on project livy-coverage-report

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-644.
--
Fix Version/s: 0.7.0
 Assignee: Yiheng Wang
   Resolution: Fixed

Issue resolved by pull request 229
https://github.com/apache/incubator-livy/pull/229

> Flaky test: Failed to execute goal 
> org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on 
> project livy-coverage-report
> 
>
> Key: LIVY-644
> URL: https://issues.apache.org/jira/browse/LIVY-644
> Project: Livy
>  Issue Type: Bug
>  Components: Tests
>Reporter: Yiheng Wang
>Assignee: Yiheng Wang
>Priority: Minor
> Fix For: 0.7.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Recently a lot of Travis job failed when generating coverage report:
> [https://travis-ci.org/apache/incubator-livy/jobs/575142847]
> [https://travis-ci.org/apache/incubator-livy/jobs/561700903]
> [https://travis-ci.org/apache/incubator-livy/jobs/508574433]
> [https://travis-ci.org/apache/incubator-livy/jobs/508574435]
> [https://travis-ci.org/apache/incubator-livy/jobs/508066760]
> [https://travis-ci.org/apache/incubator-livy/jobs/507989073]
> [https://travis-ci.org/apache/incubator-livy/jobs/574702251]
> [https://travis-ci.org/apache/incubator-livy/jobs/574686891]
> [https://travis-ci.org/apache/incubator-livy/jobs/574363881]
> [https://travis-ci.org/apache/incubator-livy/jobs/574215174]
> [https://travis-ci.org/apache/incubator-livy/jobs/573689926]
>  
> Here is the error stack:
>  
> [ERROR] Failed to execute goal 
> org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on 
> project livy-coverage-report: An error has occurred in JaCoCo Aggregate 
> report generation. Error while creating report: null: EOFException -> [Help 1]
> 2988org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.jacoco:jacoco-maven-plugin:0.8.2:report-aggregate (jacoco-report) on 
> project livy-coverage-report: An error has occurred in JaCoCo Aggregate 
> report generation.
> 2989at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:213)
> 2990at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:154)
> 2991at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:146)
> 2992at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:117)
> 2993at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:81)
> 2994at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>  (SingleThreadedBuilder.java:51)
> 2995at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
> (LifecycleStarter.java:128)
> 2996at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
> 2997at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
> 2998at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
> 2999at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
> 3000at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
> 3001at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
> 3002at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> 3003at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
> 3004at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
> 3005at java.lang.reflect.Method.invoke (Method.java:498)
> 3006at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
> (Launcher.java:289)
> 3007at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
> (Launcher.java:229)
> 3008at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
> (Launcher.java:415)
> 3009at org.codehaus.plexus.classworlds.launcher.Launcher.main 
> (Launcher.java:356)
> 3010Caused by: org.apache.maven.plugin.MojoExecutionException: An error has 
> occurred in JaCoCo Aggregate report generation.
> 3011at org.jacoco.maven.AbstractReportMojo.execute 
> (AbstractReportMojo.java:167)
> 3012at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
> (DefaultBuildPluginManager.java:134)
> 3013at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:208)
> 3014at org.apache.maven.lifecycle.internal.MojoEx

Re: Exception in SASL negotiation

[Saisai Shao]

Hi,

We don't have a test on Windows Linux subsystem, so I'm not exactly sure
the issue is, why don't you open a JIRA ticket about this problem, so that
we could track and fix it.

Thanks
Saisai

César Tenganán  于2019年9月18日周三 上午12:14写道：

> Hi,
>
> We have been working to configure Apache livy-0.5.0-incubating with Spark
> on Windows,  in this case using WSL Windows Subsystem for Linux /
> Ubuntu-18.04 distribution.
> Both Livy and spark have been configured on the linux subsystem, but Livy
> is throwing an error when it is creating the spark session that says:
>
> 19/09/17 09:51:41 INFO RpcServer$SaslServerHandler: Exception in SASL
> negotiation.
> java.lang.IllegalArgumentException: Unexpected client ID
> 'fca3ee25-ae81-42a7-b07d-9613238bd820' in SASL handshake.
> at org.apache.livy.rsc.Utils.checkArgument(Utils.java:40)
> at
> org.apache.livy.rsc.rpc.RpcServer$SaslServerHandler.update(RpcServer.java:288)
> at org.apache.livy.rsc.rpc.SaslHandler.channelRead0(SaslHandler.java:61)
> at org.apache.livy.rsc.rpc.SaslHandler.channelRead0(SaslHandler.java:36)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
> at
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
> at
> io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
> at java.lang.Thread.run(Thread.java:748)
> 19/09/17 09:51:41 WARN DefaultChannelPipeline: An exceptionCaught() event
> was fired, and it reached at the tail of the pipeline. It usually means the
> last handler in the pipeline did not handle the exception.
> java.lang.IllegalArgumentException: Unexpected client ID
> 'fca3ee25-ae81-42a7-b07d-9613238bd820' in SASL handshake.
> at org.apache.livy.rsc.Utils.checkArgument(Utils.java:40)
> at
> org.apache.livy.rsc.rpc.RpcServer$SaslServerHandler.update(RpcServer.java:288)
> at org.apache.livy.rsc.rpc.SaslHandler.channelRead0(SaslHandler.java:61)
> at org.apache.livy.rsc.rpc.SaslHandler.channelRead0(SaslHandler.java:36)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>
> We have tested the same configuration directly on Ubuntu, Centos and MacOS
> machines and it worked correctly, the error just appeared when we try on
> Windows Subsystem Linux / Ubuntu-18.04 distro.
>
> Could you please help us to understand what this error means? or how can
> we validate that the system has the requirements to perform the SASL
> negotiation correctly?
>
> I have attached a file with more details of trace log
>
> Thanks for your help!
>
> --
> Julio César Tenganán Daza
> Software Engineer
>

[jira] [Resolved] (SPARK-29112) Expose more details when ApplicationMaster reporter faces a fatal exception

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-29112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved SPARK-29112.
-
Fix Version/s: 3.0.0
 Assignee: Lantao Jin
   Resolution: Fixed

Issue resolved by pull request 25810
https://github.com/apache/spark/pull/25810

> Expose more details when ApplicationMaster reporter faces a fatal exception
> ---
>
> Key: SPARK-29112
> URL: https://issues.apache.org/jira/browse/SPARK-29112
> Project: Spark
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 2.4.4, 3.0.0
>Reporter: Lantao Jin
>Assignee: Lantao Jin
>Priority: Minor
> Fix For: 3.0.0
>
>
> In {{ApplicationMaster.Reporter}} thread, fatal exception information is 
> swallowed. It's better to expose.
> A thrift server was shutdown due to some fatal exception but no useful 
> information from log.
> {code}
> 19/09/16 06:59:54,498 INFO [Reporter] yarn.ApplicationMaster:54 : Final app 
> status: FAILED, exitCode: 12, (reason: Exception was thrown 1 time(s) from 
> Reporter thread.)
> 19/09/16 06:59:54,500 ERROR [Driver] thriftserver.HiveThriftServer2:91 : 
> Error starting HiveThriftServer2
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at 
> org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:160)
> at 
> org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:708)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (LIVY-657) Travis failed on should not create sessions with duplicate names

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-657.
--
Fix Version/s: 0.7.0
 Assignee: runzhiwang
   Resolution: Fixed

Issue resolved by pull request 225
https://github.com/apache/incubator-livy/pull/225

> Travis failed on should not create sessions with duplicate names
> 
>
> Key: LIVY-657
> URL: https://issues.apache.org/jira/browse/LIVY-657
> Project: Livy
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.6.0
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> should not create sessions with duplicate names *** FAILED *** (17 
> milliseconds)
>  session2.stopped was false (SessionManagerSpec.scala:96)
>  
> please reference to https://travis-ci.org/apache/incubator-livy/jobs/579604782



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (LIVY-666) Support named interpreter groups

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931976#comment-16931976
 ] 

Saisai Shao commented on LIVY-666:
--

I think it looks good to me. With the above, I think the REST APIs should also 
be updated, especially statements related ones, would you please propose the 
changes here, mainly about the compatibility.

> Support named interpreter groups
> 
>
> Key: LIVY-666
> URL: https://issues.apache.org/jira/browse/LIVY-666
> Project: Livy
>  Issue Type: New Feature
>Reporter: Naman Mishra
>Priority: Major
> Attachments: multiple_interpreter_groups.png
>
>
> Currently, a session can contain only one interpreter group. In order to 
> support use case of multiple repls with the same spark application multiple 
> interpreters with different variable scoping (something similar to scoped 
> interpreter mode in Zeppelin: 
> [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode
>  
> |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]),
>  I propose to have "named interpreter groups", i.e., multiple interpreter 
> groups in a session all sharing a spark context. The interpreter group can be 
> specified on which the execution is supposed to happen in the execution API.
> Similar ask has been put in LIVY-325



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (LIVY-633) session should not be gc-ed for long running queries

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-633.
--
Fix Version/s: 0.7.0
 Assignee: Yiheng Wang
   Resolution: Fixed

Issue resolved by pull request 224
https://github.com/apache/incubator-livy/pull/224

> session should not be gc-ed for long running queries
> 
>
> Key: LIVY-633
> URL: https://issues.apache.org/jira/browse/LIVY-633
> Project: Livy
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 0.6.0
>Reporter: Liju
>Assignee: Yiheng Wang
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> If you have set a relatively small session timeout eg 15 mins and query 
> execution is taking > 15 mins , the session gets gc-ed , which is incorrect 
> wrt user experience as the user was still active on session and waiting for 
> result



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (SPARK-29114) Dataset.coalesce(10) throw ChunkFetchFailureException when original Dataset size is big

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-29114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated SPARK-29114:

Priority: Major  (was: Blocker)

> Dataset.coalesce(10) throw ChunkFetchFailureException when original 
> Dataset size is big
> 
>
> Key: SPARK-29114
> URL: https://issues.apache.org/jira/browse/SPARK-29114
> Project: Spark
>  Issue Type: Bug
>  Components: Block Manager
>Affects Versions: 2.3.0
>Reporter: ZhanxiongWang
>Priority: Major
>
> I create a Dataset df with 200 partitions. I applied for 100 executors 
> for my task. Each executor with 1 core, and driver memory is 8G executor is 
> 16G. I use df.cache() before df.coalesce(10). When{color:#de350b} 
> Dataset{color} {color:#de350b}size is small{color}, the program works 
> well. But when I {color:#de350b}increase{color} the size of the Dataset, 
> the function {color:#de350b}df.coalesce(10){color} will throw 
> ChunkFetchFailureException.
> 19/09/17 08:26:44 INFO CoarseGrainedExecutorBackend: Got assigned task 210
> 19/09/17 08:26:44 INFO Executor: Running task 0.0 in stage 3.0 (TID 210)
> 19/09/17 08:26:44 INFO MapOutputTrackerWorker: Updating epoch to 1 and 
> clearing cache
> 19/09/17 08:26:44 INFO TorrentBroadcast: Started reading broadcast variable 
> 1003
> 19/09/17 08:26:44 INFO MemoryStore: Block broadcast_1003_piece0 stored as 
> bytes in memory (estimated size 49.4 KB, free 3.8 GB)
> 19/09/17 08:26:44 INFO TorrentBroadcast: Reading broadcast variable 1003 took 
> 7 ms
> 19/09/17 08:26:44 INFO MemoryStore: Block broadcast_1003 stored as values in 
> memory (estimated size 154.5 KB, free 3.8 GB)
> 19/09/17 08:26:44 INFO BlockManager: Found block rdd_1005_0 locally
> 19/09/17 08:26:44 INFO BlockManager: Found block rdd_1005_1 locally
> 19/09/17 08:26:44 INFO TransportClientFactory: Successfully created 
> connection to /100.76.29.130:54238 after 1 ms (0 ms spent in bootstraps)
> 19/09/17 08:26:46 ERROR RetryingBlockFetcher: Failed to fetch block 
> rdd_1005_18, and will not retry (0 retries)
> org.apache.spark.network.client.ChunkFetchFailureException: Failure while 
> fetching StreamChunkId\{streamId=69368607002, chunkIndex=0}: readerIndex: 0, 
> writerIndex: -2137154997 (expected: 0 <= readerIndex <= writerIndex <= 
> capacity(-2137154997))
>  at 
> org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:182)
>  at 
> org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:120)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
>  at 
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
>  at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
>  at 
> org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
>  at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:962)
>  at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
>  at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
>  at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:485)
>  at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:399)
>  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371)
>  at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
>  at 
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>  at java.lang.Thread.run(Thread.java:74

[jira] [Resolved] (LIVY-647) Travis failed on "batch session should not be gc-ed until application is finished"

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-647.
--
Fix Version/s: 0.7.0
 Assignee: runzhiwang
   Resolution: Fixed

Issue resolved by pull request 222
https://github.com/apache/incubator-livy/pull/222

> Travis failed on "batch session should not be gc-ed until application is 
> finished"
> --
>
> Key: LIVY-647
> URL: https://issues.apache.org/jira/browse/LIVY-647
> Project: Livy
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 0.6.0
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> - batch session should not be gc-ed until application is finished *** FAILED 
> *** (50 milliseconds)
>  org.mockito.exceptions.misusing.UnfinishedStubbingException: Unfinished 
> stubbing detected here:
>  -> at 
> org.apache.livy.sessions.SessionManagerSpec$$anonfun$1.org$apache$livy$sessions$SessionManagerSpec$$anonfun$$changeStateAndCheck$1(SessionManagerSpec.scala:129)
>  E.g. thenReturn() may be missing.
>  Examples of correct stubbing:
>  when(mock.isOk()).thenReturn(true);
>  when(mock.isOk()).thenThrow(exception);
>  doThrow(exception).when(mock).someVoidMethod();
>  Hints:
>  1. missing thenReturn()
>  2. you are trying to stub a final method, you naughty developer!
>  at 
> org.apache.livy.sessions.SessionManagerSpec$$anonfun$1.org$apache$livy$sessions$SessionManagerSpec$$anonfun$$changeStateAndCheck$1(SessionManagerSpec.scala:129)
>  at 
> org.apache.livy.sessions.SessionManagerSpec$$anonfun$1$$anonfun$org$apache$livy$sessions$SessionManagerSpec$$anonfun$$testSessionGC$1$1.apply(SessionManagerSpec.scala:141)
>  at 
> org.apache.livy.sessions.SessionManagerSpec$$anonfun$1$$anonfun$org$apache$livy$sessions$SessionManagerSpec$$anonfun$$testSessionGC$1$1.apply(SessionManagerSpec.scala:135)
>  at scala.collection.immutable.List.foreach(List.scala:392)
>  at 
> org.apache.livy.sessions.SessionManagerSpec$$anonfun$1.org$apache$livy$sessions$SessionManagerSpec$$anonfun$$testSessionGC$1(SessionManagerSpec.scala:135)
>  at 
> org.apache.livy.sessions.SessionManagerSpec$$anonfun$1$$anonfun$apply$mcV$sp$8.apply$mcV$sp(SessionManagerSpec.scala:108)
>  at 
> org.apache.livy.sessions.SessionManagerSpec$$anonfun$1$$anonfun$apply$mcV$sp$8.apply(SessionManagerSpec.scala:98)
>  at 
> org.apache.livy.sessions.SessionManagerSpec$$anonfun$1$$anonfun$apply$mcV$sp$8.apply(SessionManagerSpec.scala:98)
>  at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>  at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>  ...
>  
> please reference to 
> https://travis-ci.org/runzhiwang/incubator-livy/builds/575627338



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (LIVY-325) Refactoring Livy Session and Interpreter

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928479#comment-16928479
 ] 

Saisai Shao commented on LIVY-325:
--

I don't have a plan on it, please go ahead if you would like to do it. But 
since this is a large work, please do have a good plan and design doc about it, 
also separate down the work to small ones.

> Refactoring Livy Session and Interpreter
> 
>
> Key: LIVY-325
> URL: https://issues.apache.org/jira/browse/LIVY-325
> Project: Livy
>  Issue Type: New Feature
>  Components: REPL
>Affects Versions: 0.4.0
>Reporter: Saisai Shao
>Priority: Major
>
> Currently in Livy master code, Livy interpreter is bound with Livy session, 
> when we created a session, we should specify which interpreter we want, and 
> this interpreter will be created implicitly. This potentially has several 
> limitations:
> 1. We cannot create a share session, when we choose one language, we have to 
> create a new session to use. But some notebooks like Zeppelin could use 
> python, scala, R to manipulate data under the same SparkContext. So in Livy 
> we should decouple interpreter with SC and support shared context between 
> different interpreters.
> 2. Furthermore, we cannot create multiple same interpreters in one session. 
> For example in Zeppelin scope mode, it could create multiple scala 
> interpreters to share with one context, but unfortunately in current Livy we 
> could not support this.
> So based on the problems we mentioned above, we mainly have three things:
> 1. Decouple interpreters from Spark context, so that we could create multiple 
> interpreters under one context.
> 2. Make sure multiple interpreters could be worked together.
> 3. Change REST APIs to support multiple interpreters per session.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927265#comment-16927265
 ] 

Saisai Shao commented on SPARK-29038:
-

[~cltlfcjin] I think we need a SPIP review and vote on the dev mail list before 
starting the works.

> SPIP: Support Spark Materialized View
> -
>
> Key: SPARK-29038
> URL: https://issues.apache.org/jira/browse/SPARK-29038
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Affects Versions: 3.0.0
>Reporter: Lantao Jin
>Priority: Major
>
> Materialized view is an important approach in DBMS to cache data to 
> accelerate queries. By creating a materialized view through SQL, the data 
> that can be cached is very flexible, and needs to be configured arbitrarily 
> according to specific usage scenarios. The Materialization Manager 
> automatically updates the cache data according to changes in detail source 
> tables, simplifying user work. When user submit query, Spark optimizer 
> rewrites the execution plan based on the available materialized view to 
> determine the optimal execution plan.
> Details in [design 
> doc|https://docs.google.com/document/d/1q5pjSWoTNVc9zsAfbNzJ-guHyVwPsEroIEP8Cca179A/edit?usp=sharing]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927263#comment-16927263
 ] 

Saisai Shao commented on SPARK-29038:
-

IIUC, I think the key difference between MV and Spark's built-in {{CACHE}} 
support is: 1. MV needs update when source table is updated, which I think 
current Spark's {{CACHE}} cannot support; 2. classical MV requires writing of 
source query based on the existing MV, which I think current Spark doesn't 
have. Please correct me if I'm wrong.

> SPIP: Support Spark Materialized View
> -
>
> Key: SPARK-29038
> URL: https://issues.apache.org/jira/browse/SPARK-29038
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Affects Versions: 3.0.0
>Reporter: Lantao Jin
>Priority: Major
>
> Materialized view is an important approach in DBMS to cache data to 
> accelerate queries. By creating a materialized view through SQL, the data 
> that can be cached is very flexible, and needs to be configured arbitrarily 
> according to specific usage scenarios. The Materialization Manager 
> automatically updates the cache data according to changes in detail source 
> tables, simplifying user work. When user submit query, Spark optimizer 
> rewrites the execution plan based on the available materialized view to 
> determine the optimal execution plan.
> Details in [design 
> doc|https://docs.google.com/document/d/1q5pjSWoTNVc9zsAfbNzJ-guHyVwPsEroIEP8Cca179A/edit?usp=sharing]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: Welcoming some new committers and PMC members

[Saisai Shao]

Congratulations!

Jungtaek Lim  于2019年9月9日周一 下午6:11写道：

> Congratulations! Well deserved!
>
> On Tue, Sep 10, 2019 at 9:51 AM John Zhuge  wrote:
>
>> Congratulations!
>>
>> On Mon, Sep 9, 2019 at 5:45 PM Shane Knapp  wrote:
>>
>>> congrats everyone!  :)
>>>
>>> On Mon, Sep 9, 2019 at 5:32 PM Matei Zaharia 
>>> wrote:
>>> >
>>> > Hi all,
>>> >
>>> > The Spark PMC recently voted to add several new committers and one PMC
>>> member. Join me in welcoming them to their new roles!
>>> >
>>> > New PMC member: Dongjoon Hyun
>>> >
>>> > New committers: Ryan Blue, Liang-Chi Hsieh, Gengliang Wang, Yuming
>>> Wang, Weichen Xu, Ruifeng Zheng
>>> >
>>> > The new committers cover lots of important areas including ML, SQL,
>>> and data sources, so it’s great to have them here. All the best,
>>> >
>>> > Matei and the Spark PMC
>>> >
>>> >
>>> > -
>>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>> >
>>>
>>>
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>>
>> --
>> John Zhuge
>>
>
>
> --
> Name : Jungtaek Lim
> Blog : http://medium.com/@heartsavior
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>

[jira] [Resolved] (LIVY-659) Travis failed on "can kill spark-submit while it's running"

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-659.
--
Fix Version/s: 0.7.0
 Assignee: runzhiwang
   Resolution: Fixed

Issue resolved by pull request 226
https://github.com/apache/incubator-livy/pull/226

> Travis failed on "can kill spark-submit while it's running"
> ---
>
> Key: LIVY-659
> URL: https://issues.apache.org/jira/browse/LIVY-659
> Project: Livy
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.6.0
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> * can kill spark-submit while it's running *** FAILED *** (41 milliseconds)
>  org.mockito.exceptions.verification.WantedButNotInvoked: Wanted but not 
> invoked:
> lineBufferedProcess.destroy();
> -> at 
> org.apache.livy.utils.SparkYarnAppSpec$$anonfun$1$$anonfun$apply$mcV$sp$13$$anonfun$apply$mcV$sp$15$$anonfun$apply$mcV$sp$16.apply$mcV$sp(SparkYarnAppSpec.scala:226)
> Actually, there were zero interactions with this mock.
>  at 
> org.apache.livy.utils.SparkYarnAppSpec$$anonfun$1$$anonfun$apply$mcV$sp$13$$anonfun$apply$mcV$sp$15$$anonfun$apply$mcV$sp$16.apply$mcV$sp(SparkYarnAppSpec.scala:226)
>  at 
> org.apache.livy.utils.SparkYarnAppSpec.org$apache$livy$utils$SparkYarnAppSpec$$cleanupThread(SparkYarnAppSpec.scala:43)
>  at 
> org.apache.livy.utils.SparkYarnAppSpec$$anonfun$1$$anonfun$apply$mcV$sp$13$$anonfun$apply$mcV$sp$15.apply$mcV$sp(SparkYarnAppSpec.scala:224)
>  at org.apache.livy.utils.Clock$.withSleepMethod(Clock.scala:31)
>  at 
> org.apache.livy.utils.SparkYarnAppSpec$$anonfun$1$$anonfun$apply$mcV$sp$13.apply$mcV$sp(SparkYarnAppSpec.scala:201)
>  at 
> org.apache.livy.utils.SparkYarnAppSpec$$anonfun$1$$anonfun$apply$mcV$sp$13.apply(SparkYarnAppSpec.scala:201)
>  at 
> org.apache.livy.utils.SparkYarnAppSpec$$anonfun$1$$anonfun$apply$mcV$sp$13.apply(SparkYarnAppSpec.scala:201)
>  at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>  at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> please reference to: 
> https://travis-ci.org/captainzmc/incubator-livy/jobs/580596561



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Question about schema evolution and partition spec evolution

[Saisai Shao]

Hi team,

I have some newbie questions about schema evolution and partition
evolution. From the design spec, Iceberg supports schema evolution and
partition spec evolution, my questions are:

1. If a new column is added, are we going to rewrite the whole data, if not
how do we support it?
2. Do we support partition spec evolution to add new partition column? If
so, does it require data rewrite, since the directories may be different?
3. Do we support changing partition strategy during partition spec
evolution, say from identity to bucket? If so, I think it requires data
rewrite, am I correct? Also do we need to keep old data, so that historical
revisit will get a correct result.

Sorry about newbie questions, since Iceberg is a mutable table, which makes
problem more complicated, and I'm doing bucketing support, so I'm thinking
about schema evolution which potentially affects bucketing.

Best regards,
Saisai

[jira] [Resolved] (LIVY-645) Add Name, Owner, Proxy User to web UI

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-645.
--
Fix Version/s: 0.7.0
 Assignee: Jeffrey(Xilang) Yan
   Resolution: Fixed

Issue resolved by pull request 207
https://github.com/apache/incubator-livy/pull/207

> Add Name, Owner, Proxy User to web UI
> -
>
> Key: LIVY-645
> URL: https://issues.apache.org/jira/browse/LIVY-645
> Project: Livy
>  Issue Type: Improvement
>  Components: Server
>Affects Versions: 0.6.0
>Reporter: Jeffrey(Xilang) Yan
>Assignee: Jeffrey(Xilang) Yan
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> In current web UI, Interactive Sessions list has no Name, Batch Sessions list 
> has no Name, Owner and Proxy User. Should add those information so user can 
> find their session quickly.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Reopened] (SPARK-19147) netty throw NPE

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-19147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao reopened SPARK-19147:
-

> netty throw NPE
> ---
>
> Key: SPARK-19147
> URL: https://issues.apache.org/jira/browse/SPARK-19147
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: cen yuhai
>Priority: Major
>  Labels: bulk-closed
>
> {code}
> 17/01/10 19:17:20 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) 
> from bigdata-hdp-apache1828.xg01.diditaxi.com:7337
> java.lang.NullPointerException: group
>   at io.netty.bootstrap.AbstractBootstrap.group(AbstractBootstrap.java:80)
>   at 
> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:203)
>   at 
> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:181)
>   at 
> org.apache.spark.network.shuffle.ExternalShuffleClient$1.createAndStart(ExternalShuffleClient.java:105)
>   at 
> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
>   at 
> org.apache.spark.network.shuffle.RetryingBlockFetcher.start(RetryingBlockFetcher.java:120)
>   at 
> org.apache.spark.network.shuffle.ExternalShuffleClient.fetchBlocks(ExternalShuffleClient.java:114)
>   at 
> org.apache.spark.storage.ShuffleBlockFetcherIterator.sendRequest(ShuffleBlockFetcherIterator.scala:169)
>   at 
> org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchUpToMaxBytes(ShuffleBlockFetcherIterator.scala:354)
>   at 
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)
>   at 
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)
>   at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
>   at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
>   at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>   at 
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$2.hasNext(WholeStageCodegenExec.scala:396)
>   at 
> org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:138)
>   at 
> org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:215)
>   at 
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:957)
>   at 
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
>   at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888)
>   at 
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948)
>   at 
> org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694)
>   at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
>   at org.apache.

[jira] [Commented] (SPARK-19147) netty throw NPE

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-19147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923014#comment-16923014
 ] 

Saisai Shao commented on SPARK-19147:
-

Yes, we also met the similar issue when executor is stopping, floods of netty 
NPE appears. I'm going to reopen this issue, at least we should improve the 
exception message. 

> netty throw NPE
> ---
>
> Key: SPARK-19147
> URL: https://issues.apache.org/jira/browse/SPARK-19147
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: cen yuhai
>Priority: Major
>  Labels: bulk-closed
>
> {code}
> 17/01/10 19:17:20 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) 
> from bigdata-hdp-apache1828.xg01.diditaxi.com:7337
> java.lang.NullPointerException: group
>   at io.netty.bootstrap.AbstractBootstrap.group(AbstractBootstrap.java:80)
>   at 
> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:203)
>   at 
> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:181)
>   at 
> org.apache.spark.network.shuffle.ExternalShuffleClient$1.createAndStart(ExternalShuffleClient.java:105)
>   at 
> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
>   at 
> org.apache.spark.network.shuffle.RetryingBlockFetcher.start(RetryingBlockFetcher.java:120)
>   at 
> org.apache.spark.network.shuffle.ExternalShuffleClient.fetchBlocks(ExternalShuffleClient.java:114)
>   at 
> org.apache.spark.storage.ShuffleBlockFetcherIterator.sendRequest(ShuffleBlockFetcherIterator.scala:169)
>   at 
> org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchUpToMaxBytes(ShuffleBlockFetcherIterator.scala:354)
>   at 
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)
>   at 
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)
>   at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
>   at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
>   at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>   at 
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
>  Source)
>   at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>   at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>   at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$2.hasNext(WholeStageCodegenExec.scala:396)
>   at 
> org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:138)
>   at 
> org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:215)
>   at 
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:957)
>   at 
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
>   at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888)
>   at 
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948)
>   at 
> org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694)
>   at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
>   at org.apache.spa

[jira] [Resolved] (LIVY-652) Thrifserver doesn't set session name correctly

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-652.
--
Fix Version/s: 0.7.0
 Assignee: Jeffrey(Xilang) Yan
   Resolution: Fixed

Issue resolved by pull request 218
https://github.com/apache/incubator-livy/pull/218

> Thrifserver doesn't set session name correctly
> --
>
> Key: LIVY-652
> URL: https://issues.apache.org/jira/browse/LIVY-652
> Project: Livy
>  Issue Type: Bug
>  Components: Thriftserver
>Affects Versions: 0.6.0
>Reporter: Jeffrey(Xilang) Yan
>Assignee: Jeffrey(Xilang) Yan
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Thrift server doesn't set session name correctly, so not able to view session 
> name in Livy Web UI



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (LIVY-519) Flaky test: SparkYarnApp "should kill yarn app "

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-519.
--
Fix Version/s: 0.7.0
 Assignee: runzhiwang
   Resolution: Fixed

Issue resolved by pull request 221
https://github.com/apache/incubator-livy/pull/221

> Flaky test: SparkYarnApp "should kill yarn app "
> 
>
> Key: LIVY-519
> URL: https://issues.apache.org/jira/browse/LIVY-519
> Project: Livy
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.6.0
>Reporter: Marcelo Vanzin
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> From a travis run:
> {noformat}
> SparkYarnApp
> - should poll YARN state and terminate (116 milliseconds)
> - should kill yarn app *** FAILED *** (83 milliseconds)
>   org.mockito.exceptions.verification.WantedButNotInvoked: Wanted but 
> not invoked:
> yarnClient.killApplication(
> application_1467912463905_0021
> );
> -> at 
> org.apache.livy.utils.SparkYarnAppSpec$$anonfun$1$$anonfun$apply$mcV$sp$5$$anonfun$apply$mcV$sp$7$$anonfun$apply$mcV$sp$8.apply$mcV$sp(SparkYarnAppSpec.scala:156)
> 
> However, there were other interactions with this mock:
> -> at 
> org.apache.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:261)
> -> at 
> org.apache.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:270)
>   at 
> org.apache.livy.utils.SparkYarnAppSpec$$anonfun$1$$anonfun$apply$mcV$sp$5$$anonfun$apply$mcV$sp$7$$anonfun$apply$mcV$sp$8.apply$mcV$sp(SparkYarnAppSpec.scala:156)
>   at 
> org.apache.livy.utils.SparkYarnAppSpec.org$apache$livy$utils$SparkYarnAppSpec$$cleanupThread(SparkYarnAppSpec.scala:43)
>   at 
> org.apache.livy.utils.SparkYarnAppSpec$$anonfun$1$$anonfun$apply$mcV$sp$5$$anonfun$apply$mcV$sp$7.apply$mcV$sp(SparkYarnAppSpec.scala:148)
>   at org.apache.livy.utils.Clock$.withSleepMethod(Clock.scala:31)
>   at 
> org.apache.livy.utils.SparkYarnAppSpec$$anonfun$1$$anonfun$apply$mcV$sp$5.apply$mcV$sp(SparkYarnAppSpec.scala:126)
>   at 
> org.apache.livy.utils.SparkYarnAppSpec$$anonfun$1$$anonfun$apply$mcV$sp$5.apply(SparkYarnAppSpec.scala:126)
>   at 
> org.apache.livy.utils.SparkYarnAppSpec$$anonfun$1$$anonfun$apply$mcV$sp$5.apply(SparkYarnAppSpec.scala:126)
>   at 
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>   ...
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: New committer and PPMC member, Anton Okolnychyi

[Saisai Shao]

Congrats Anton!

Best regards,
Saisai

Daniel Weeks  于2019年9月3日周二 上午7:48写道：

> Congrats Anton!
>
> On Fri, Aug 30, 2019 at 1:54 PM Edgar Rodriguez
>  wrote:
>
>> Nice! Congratulations, Anton!
>>
>> Cheers,
>>
>> On Fri, Aug 30, 2019 at 1:42 PM Dongjoon Hyun 
>> wrote:
>>
>>> Congratulations, Anton! :D
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>> On Fri, Aug 30, 2019 at 10:06 AM Ryan Blue  wrote:
>>>
 I'd like to congratulate Anton Okolnychyi, who was just invited to join
 the Iceberg committers and PPMC!

 Thanks for all your contributions, Anton!

 rb

 --
 Ryan Blue

>>>
>>
>> --
>> Edgar Rodriguez
>>
>

Re: Possible missing mentor(s)

[Saisai Shao]

First, we're scaling the PR review, but we only have few active committers,
so the merge may not be fast.
Second, if someone has a *good* and *large* contribution history, and
actively participates in community, we will add him without doubt.
Third, 2-year old open PRs doesn't stand anything, some reviewers left the
community and PRs get staled, it is quite common, especially in large
community.

Sid Anand  于2019年9月1日周日 下午4:46写道：

> Apache projects promote contributors to committers based on contributions
> made, not on an expectation of future activity. That's the Apache way per
> my understanding. Over time, folks become inactive and busy -- life
> happens, I get it.  May I ask what are you folks doing to scale PR review
> and merging? Are you adding new committers?  Do you feel that 2-year old
> open PRs is where you wish to be and is the right way to grow a community?
>
> On Sun, Sep 1, 2019 at 1:46 AM Sid Anand  wrote:
>
> > Apache projects promote contributors to committers based on contributions
> > made, not on an expectation of future activity. That's the Apache way per
> > my understanding. Over time, folks become inactive and busy -- life
> > happens, I get it.  May I ask what are you folks doing to scale PR review
> > and merging? Are you adding new committers?  Do you feel that 2-year old
> > open PRs is where you wish to be and is the right way to grow a
> community?
> >
> > -s
> >
> > On Sun, Sep 1, 2019 at 12:59 AM Saisai Shao 
> > wrote:
> >
> >> It's unfair to say there's underlying bias. Livy project is a small
> >> project, the contributor diversity may not be as rich as popular project
> >> like Spark, it is not fair to say that the contributions only limits to
> >> someones, so project is biased. There're many small Apache project which
> >> has only few contributors, can we say those projects are biased? Also
> for
> >> years the committers have joined and left the community, it is hard to
> >> track every contribution in time, as we're not a full-time Livy open
> >> source
> >> contributors. I also have several PRs left unreviewed for years. It's
> >> quite
> >> common even for large project like Spark, Hadoop, there're so many
> >> un-merged PRs left for several years. It's unfair to say the project is
> >> biased, unhealthy because of some un-merged PRs.
> >>
> >> The community is small but free and open, I would deny that the
> community
> >> is unhealthy especially biased, this is an irresponsible and subjective
> >> word.
> >>
> >> Sid Anand  于2019年9月1日周日 上午4:20写道：
> >>
> >> > Folks!
> >> > We've (several devs, myself included) contacted the livy dev list and
> >> the
> >> > owners DL several times. Our PRs stagnated over a few years. Livy is a
> >> > central component in PayPal's Data Infra (Our data footprint is 80+
> PB).
> >> > The project seems pretty unhealthy. After a few years, this dev moved
> on
> >> > and the state of our PR may be harder to define, with both absentee
> >> > mentors/PMC/committers and PR author.
> >> >
> >> > I see only a narrow band of contributors being merged. I hope there is
> >> no
> >> > underlying bias, given that this would not be the Apache way. As
> >> mentors, I
> >> > hope the goal is to watch for such bias and eliminate it to promote
> >> > community health, engagement, and integrity.
> >> >
> >> > -s
> >> >
> >> > On Fri, Aug 30, 2019 at 10:22 PM Jean-Baptiste Onofré <
> j...@nanthrax.net>
> >> > wrote:
> >> >
> >> > > Hi Justin,
> >> > >
> >> > > Like Luciano, I'm also around. I've proposed couple of new features
> >> that
> >> > > I plan to work on and I try to review/verify the releases.
> >> > >
> >> > > Regards
> >> > > JB
> >> > >
> >> > > On 21/10/2018 01:50, Justin Mclean wrote:
> >> > > > Hi,
> >> > > >
> >> > > > We have discussed missing mentors on the incubator general list,
> >> > > identified those who may be missing and over a couple of months
> tried
> >> to
> >> > > contact your mentor in several ways but have got no reply so they
> have
> >> > been
> >> > > removed from the roster.
> >> > > >
> >> > > > If this is error, and your mentor is actually active, please
> >> contact me
> >> > > and I'll add them back to the roster.
> >> > > >
> >> > > > The IPMC will also do what it can to find extra mentors for those
> >> > > podling that need them.
> >> > > >
> >> > > > Thanks,
> >> > > > Justin
> >> > > > (V.P. Incubator)
> >> > > >
> >> > >
> >> > > --
> >> > > Jean-Baptiste Onofré
> >> > > jbono...@apache.org
> >> > > http://blog.nanthrax.net
> >> > > Talend - http://www.talend.com
> >> > >
> >> >
> >>
> >
>

Re: Possible missing mentor(s)

[Saisai Shao]

It's unfair to say there's underlying bias. Livy project is a small
project, the contributor diversity may not be as rich as popular project
like Spark, it is not fair to say that the contributions only limits to
someones, so project is biased. There're many small Apache project which
has only few contributors, can we say those projects are biased? Also for
years the committers have joined and left the community, it is hard to
track every contribution in time, as we're not a full-time Livy open source
contributors. I also have several PRs left unreviewed for years. It's quite
common even for large project like Spark, Hadoop, there're so many
un-merged PRs left for several years. It's unfair to say the project is
biased, unhealthy because of some un-merged PRs.

The community is small but free and open, I would deny that the community
is unhealthy especially biased, this is an irresponsible and subjective
word.

Sid Anand  于2019年9月1日周日 上午4:20写道：

> Folks!
> We've (several devs, myself included) contacted the livy dev list and the
> owners DL several times. Our PRs stagnated over a few years. Livy is a
> central component in PayPal's Data Infra (Our data footprint is 80+ PB).
> The project seems pretty unhealthy. After a few years, this dev moved on
> and the state of our PR may be harder to define, with both absentee
> mentors/PMC/committers and PR author.
>
> I see only a narrow band of contributors being merged. I hope there is no
> underlying bias, given that this would not be the Apache way. As mentors, I
> hope the goal is to watch for such bias and eliminate it to promote
> community health, engagement, and integrity.
>
> -s
>
> On Fri, Aug 30, 2019 at 10:22 PM Jean-Baptiste Onofré 
> wrote:
>
> > Hi Justin,
> >
> > Like Luciano, I'm also around. I've proposed couple of new features that
> > I plan to work on and I try to review/verify the releases.
> >
> > Regards
> > JB
> >
> > On 21/10/2018 01:50, Justin Mclean wrote:
> > > Hi,
> > >
> > > We have discussed missing mentors on the incubator general list,
> > identified those who may be missing and over a couple of months tried to
> > contact your mentor in several ways but have got no reply so they have
> been
> > removed from the roster.
> > >
> > > If this is error, and your mentor is actually active, please contact me
> > and I'll add them back to the roster.
> > >
> > > The IPMC will also do what it can to find extra mentors for those
> > podling that need them.
> > >
> > > Thanks,
> > > Justin
> > > (V.P. Incubator)
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>

[jira] [Resolved] (LIVY-586) When a batch fails on startup, Livy continues to report the batch as "starting", even though it has failed

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-586.
--
Fix Version/s: 0.7.0
 Assignee: runzhiwang
   Resolution: Fixed

Issue resolved by pull request 215
https://github.com/apache/incubator-livy/pull/215

> When a batch fails on startup, Livy continues to report the batch as 
> "starting", even though it has failed
> --
>
> Key: LIVY-586
> URL: https://issues.apache.org/jira/browse/LIVY-586
> Project: Livy
>  Issue Type: Bug
>  Components: Batch
>Affects Versions: 0.5.0
> Environment: AWS EMR, Livy submits batches to YARN in cluster mode
>Reporter: Sam Brougher
>Assignee: runzhiwang
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> When starting a Livy batch, I accidentally provided it a jar location in S3 
> that did not exist. Livy then continued to report that the job was 
> "starting", even though it had clearly failed.
> stdout:
> {code:java}
> 2019-04-05 11:24:18,149 [main] WARN org.apache.hadoop.util.NativeCodeLoader 
> [appName=] [jobId=] [clusterId=] - Unable to load native-hadoop library for 
> your platform... using builtin-java classes where applicable
> Warning: Skip remote jar 
> s3://dev-dp-local/jars/develop-fix/ap5-app-transform-0.2-thread-pool-SNAPSHOT.jar.
> 2019-04-05 11:24:19,152 [main] INFO org.apache.hadoop.yarn.client.RMProxy 
> [appName=] [jobId=] [clusterId=] - Connecting to ResourceManager at 
> ip-10-25-30-127.dev.cainc.internal/10.25.30.127:8032
> 2019-04-05 11:24:19,453 [main] INFO org.apache.spark.deploy.yarn.Client 
> [appName=] [jobId=] [clusterId=] - Requesting a new application from cluster 
> with 6 NodeManagers
> 2019-04-05 11:24:19,532 [main] INFO org.apache.spark.deploy.yarn.Client 
> [appName=] [jobId=] [clusterId=] - Verifying our application has not 
> requested more than the maximum memory capability of the cluster (54272 MB 
> per container)
> 2019-04-05 11:24:19,533 [main] INFO org.apache.spark.deploy.yarn.Client 
> [appName=] [jobId=] [clusterId=] - Will allocate AM container, with 9011 MB 
> memory including 819 MB overhead
> 2019-04-05 11:24:19,534 [main] INFO org.apache.spark.deploy.yarn.Client 
> [appName=] [jobId=] [clusterId=] - Setting up container launch context for 
> our AM
> 2019-04-05 11:24:19,537 [main] INFO org.apache.spark.deploy.yarn.Client 
> [appName=] [jobId=] [clusterId=] - Setting up the launch environment for our 
> AM container
> 2019-04-05 11:24:19,549 [main] INFO org.apache.spark.deploy.yarn.Client 
> [appName=] [jobId=] [clusterId=] - Preparing resources for our AM container
> 2019-04-05 11:24:21,059 [main] WARN org.apache.spark.deploy.yarn.Client 
> [appName=] [jobId=] [clusterId=] - Neither spark.yarn.jars nor 
> spark.yarn.archive is set, falling back to uploading libraries under 
> SPARK_HOME.
> 2019-04-05 11:24:23,790 [main] INFO org.apache.spark.deploy.yarn.Client 
> [appName=] [jobId=] [clusterId=] - Uploading resource 
> file:/mnt/tmp/spark-b4e4a760-77a3-4554-a3f3-c3f82675d865/__spark_libs__3639879082942366045.zip
>  -> 
> hdfs://ip-10-25-30-127.dev.cainc.internal:8020/user/livy/.sparkStaging/application_1554234858331_0222/__spark_libs__3639879082942366045.zip
> 2019-04-05 11:24:26,817 [main] INFO org.apache.spark.deploy.yarn.Client 
> [appName=] [jobId=] [clusterId=] - Uploading resource 
> s3://dev-dp-local/jars/develop-fix/ap5-app-transform-0.2-thread-pool-SNAPSHOT.jar
>  -> 
> hdfs://ip-10-25-30-127.dev.cainc.internal:8020/user/livy/.sparkStaging/application_1554234858331_0222/ap5-app-transform-0.2-thread-pool-SNAPSHOT.jar
> 2019-04-05 11:24:26,940 [main] INFO org.apache.spark.deploy.yarn.Client 
> [appName=] [jobId=] [clusterId=] - Deleted staging directory 
> hdfs://ip-10-25-30-127.dev.cainc.internal:8020/user/livy/.sparkStaging/application_1554234858331_0222
> Exception in thread "main" java.io.FileNotFoundException: No such file or 
> directory 
> 's3://dev-dp-local/jars/develop-fix/ap5-app-transform-0.2-thread-pool-SNAPSHOT.jar'
>   at 
> com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:805)
>   at 
> com.amazon.ws.emr.hadoop.fs.EmrFileSystem.getFileStatus(EmrFileSystem.java:536)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
>   at 
> org.apache.spark.deploy.yar

[jira] [Resolved] (LIVY-642) A rare status happened in yarn cause SparkApp change into error state

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-642.
--
Fix Version/s: 0.7.0
 Assignee: Zhefeng Wang
   Resolution: Fixed

Issue resolved by pull request 204
https://github.com/apache/incubator-livy/pull/204

> A rare status happened in yarn cause SparkApp change into error state
> -
>
> Key: LIVY-642
> URL: https://issues.apache.org/jira/browse/LIVY-642
> Project: Livy
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 0.6.0
>Reporter: Zhefeng Wang
>Assignee: Zhefeng Wang
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Some batch session execute successfully but return error state. in livy 
> log，we find:
> {quote}{{2019-08-08 20:11:37,678 ERROR [Logging.scala:56] - Unknown YARN 
> state RUNNING for app application_1559632632227_39801506 with final status 
> SUCCEEDED.}}
> {quote}
> and this situation with yarn state RUNNING and final status SUCCEEDED is a 
> *correct* state in yarn, so this should not mapped to SparkApp.State.FAILED



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (SPARK-28340) Noisy exceptions when tasks are killed: "DiskBlockObjectWriter: Uncaught exception while reverting partial writes to file: java.nio.channels.ClosedByInterruptException

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-28340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918386#comment-16918386
 ] 

Saisai Shao commented on SPARK-28340:
-

My simple concern is that there may be other places which will potentially 
throw this "ClosedByInterruptException" during task killing, it seems hard to 
figure out all of them.

> Noisy exceptions when tasks are killed: "DiskBlockObjectWriter: Uncaught 
> exception while reverting partial writes to file: 
> java.nio.channels.ClosedByInterruptException"
> 
>
> Key: SPARK-28340
> URL: https://issues.apache.org/jira/browse/SPARK-28340
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Josh Rosen
>Priority: Minor
>
> If a Spark task is killed while writing blocks to disk (due to intentional 
> job kills, automated killing of redundant speculative tasks, etc) then Spark 
> may log exceptions like
> {code:java}
> 19/07/10 21:31:08 ERROR storage.DiskBlockObjectWriter: Uncaught exception 
> while reverting partial writes to file /
> java.nio.channels.ClosedByInterruptException
>   at 
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>   at sun.nio.ch.FileChannelImpl.truncate(FileChannelImpl.java:372)
>   at 
> org.apache.spark.storage.DiskBlockObjectWriter$$anonfun$revertPartialWritesAndClose$2.apply$mcV$sp(DiskBlockObjectWriter.scala:218)
>   at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1369)
>   at 
> org.apache.spark.storage.DiskBlockObjectWriter.revertPartialWritesAndClose(DiskBlockObjectWriter.scala:214)
>   at 
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.stop(BypassMergeSortShuffleWriter.java:237)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:105)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
>   at org.apache.spark.scheduler.Task.run(Task.scala:121)
>   at 
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
>   at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748){code}
> If {{BypassMergeSortShuffleWriter}} is being used then a single cancelled 
> task can result in hundreds of these stacktraces being logged.
> Here are some StackOverflow questions asking about this:
>  * [https://stackoverflow.com/questions/40027870/spark-jobserver-job-crash]
>  * 
> [https://stackoverflow.com/questions/50646953/why-is-java-nio-channels-closedbyinterruptexceptio-called-when-caling-multiple]
>  * 
> [https://stackoverflow.com/questions/41867053/java-nio-channels-closedbyinterruptexception-in-spark]
>  * 
> [https://stackoverflow.com/questions/56845041/are-closedbyinterruptexception-exceptions-expected-when-spark-speculation-kills]
>  
> Can we prevent this exception from occurring? If not, can we treat this 
> "expected exception" in a special manner to avoid log spam? My concern is 
> that the presence of large numbers of spurious exceptions is confusing to 
> users when they are inspecting Spark logs to diagnose other issues.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-28340) Noisy exceptions when tasks are killed: "DiskBlockObjectWriter: Uncaught exception while reverting partial writes to file: java.nio.channels.ClosedByInterruptException

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-28340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918306#comment-16918306
 ] 

Saisai Shao commented on SPARK-28340:
-

We also saw a bunch of exceptions in our production environment. Looks like it 
is hard to prevent unless we change to not use `interrupt`, maybe we can just 
ignore logging such exceptions.

> Noisy exceptions when tasks are killed: "DiskBlockObjectWriter: Uncaught 
> exception while reverting partial writes to file: 
> java.nio.channels.ClosedByInterruptException"
> 
>
> Key: SPARK-28340
> URL: https://issues.apache.org/jira/browse/SPARK-28340
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Josh Rosen
>Priority: Minor
>
> If a Spark task is killed while writing blocks to disk (due to intentional 
> job kills, automated killing of redundant speculative tasks, etc) then Spark 
> may log exceptions like
> {code:java}
> 19/07/10 21:31:08 ERROR storage.DiskBlockObjectWriter: Uncaught exception 
> while reverting partial writes to file /
> java.nio.channels.ClosedByInterruptException
>   at 
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>   at sun.nio.ch.FileChannelImpl.truncate(FileChannelImpl.java:372)
>   at 
> org.apache.spark.storage.DiskBlockObjectWriter$$anonfun$revertPartialWritesAndClose$2.apply$mcV$sp(DiskBlockObjectWriter.scala:218)
>   at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1369)
>   at 
> org.apache.spark.storage.DiskBlockObjectWriter.revertPartialWritesAndClose(DiskBlockObjectWriter.scala:214)
>   at 
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.stop(BypassMergeSortShuffleWriter.java:237)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:105)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
>   at org.apache.spark.scheduler.Task.run(Task.scala:121)
>   at 
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
>   at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748){code}
> If {{BypassMergeSortShuffleWriter}} is being used then a single cancelled 
> task can result in hundreds of these stacktraces being logged.
> Here are some StackOverflow questions asking about this:
>  * [https://stackoverflow.com/questions/40027870/spark-jobserver-job-crash]
>  * 
> [https://stackoverflow.com/questions/50646953/why-is-java-nio-channels-closedbyinterruptexceptio-called-when-caling-multiple]
>  * 
> [https://stackoverflow.com/questions/41867053/java-nio-channels-closedbyinterruptexception-in-spark]
>  * 
> [https://stackoverflow.com/questions/56845041/are-closedbyinterruptexception-exceptions-expected-when-spark-speculation-kills]
>  
> Can we prevent this exception from occurring? If not, can we treat this 
> "expected exception" in a special manner to avoid log spam? My concern is 
> that the presence of large numbers of spurious exceptions is confusing to 
> users when they are inspecting Spark logs to diagnose other issues.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Assigned] (LIVY-616) Livy Server discovery

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao reassigned LIVY-616:


Assignee: Saisai Shao

> Livy Server discovery
> -
>
> Key: LIVY-616
> URL: https://issues.apache.org/jira/browse/LIVY-616
> Project: Livy
>  Issue Type: Improvement
>  Components: Server
>Reporter: Oleksandr Shevchenko
>    Assignee: Saisai Shao
>Priority: Major
> Attachments: Livy Server discovery.pdf, 
> image-2019-08-28-17-08-21-590.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently, there isn't a way to get Livy Server URI by the client without 
> setting Livy address explicitly to livy.conf. A client should set 
> "livy.server.host" variable and then get it via LivyConf. The same behavior 
> if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It 
> very inconvenient when we install Livy packages on couple nodes and don't 
> know where exactly Livy Server will be started e.g. by Ambari or Cloudera 
> Manager. Also, in this case, we need to have Livy configuration files on a 
> node where we want to get Livy address. 
> It will be very helpful if we will add Livy Server address to Zookeeper and 
> expose API for clients to get Livy URL to use it in client code for REST 
> calls. 
> Livy already supports state saving in Zookeeper but I don't see that we store 
> Livy server address somewhere. Before starting investigating and 
> implementation I want to ask here about this.
> Please, correct me if I missed something.
> Any comments will be highly appreciated!



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (LIVY-616) Livy Server discovery

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917755#comment-16917755
 ] 

Saisai Shao commented on LIVY-616:
--

[~oshevchenko] we don't typically assign the jira ourselves, committers will 
assign to whom the PR is finally merged.

> Livy Server discovery
> -
>
> Key: LIVY-616
> URL: https://issues.apache.org/jira/browse/LIVY-616
> Project: Livy
>  Issue Type: Improvement
>  Components: Server
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: Livy Server discovery.pdf
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently, there isn't a way to get Livy Server URI by the client without 
> setting Livy address explicitly to livy.conf. A client should set 
> "livy.server.host" variable and then get it via LivyConf. The same behavior 
> if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It 
> very inconvenient when we install Livy packages on couple nodes and don't 
> know where exactly Livy Server will be started e.g. by Ambari or Cloudera 
> Manager. Also, in this case, we need to have Livy configuration files on a 
> node where we want to get Livy address. 
> It will be very helpful if we will add Livy Server address to Zookeeper and 
> expose API for clients to get Livy URL to use it in client code for REST 
> calls. 
> Livy already supports state saving in Zookeeper but I don't see that we store 
> Livy server address somewhere. Before starting investigating and 
> implementation I want to ask here about this.
> Please, correct me if I missed something.
> Any comments will be highly appreciated!



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Assigned] (LIVY-616) Livy Server discovery

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao reassigned LIVY-616:


Assignee: (was: Oleksandr Shevchenko)

> Livy Server discovery
> -
>
> Key: LIVY-616
> URL: https://issues.apache.org/jira/browse/LIVY-616
> Project: Livy
>  Issue Type: Improvement
>  Components: Server
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: Livy Server discovery.pdf
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently, there isn't a way to get Livy Server URI by the client without 
> setting Livy address explicitly to livy.conf. A client should set 
> "livy.server.host" variable and then get it via LivyConf. The same behavior 
> if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It 
> very inconvenient when we install Livy packages on couple nodes and don't 
> know where exactly Livy Server will be started e.g. by Ambari or Cloudera 
> Manager. Also, in this case, we need to have Livy configuration files on a 
> node where we want to get Livy address. 
> It will be very helpful if we will add Livy Server address to Zookeeper and 
> expose API for clients to get Livy URL to use it in client code for REST 
> calls. 
> Livy already supports state saving in Zookeeper but I don't see that we store 
> Livy server address somewhere. Before starting investigating and 
> implementation I want to ask here about this.
> Please, correct me if I missed something.
> Any comments will be highly appreciated!



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (LIVY-617) Livy session leak on Yarn when creating session duplicated names

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-617.
--
Fix Version/s: 0.7.0
 Assignee: shanyu zhao
   Resolution: Fixed

Issue resolved by pull request 187
https://github.com/apache/incubator-livy/pull/187

> Livy session leak on Yarn when creating session duplicated names
> 
>
> Key: LIVY-617
> URL: https://issues.apache.org/jira/browse/LIVY-617
> Project: Livy
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 0.6.0
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> When running Livy on Yarn and try to create session with duplicated names, 
> Livy server sends response to client "Duplicate session name: xxx" but it 
> doesn't stop the session. The session creation failed, however, the Yarn 
> application got started and keeps running forever.
> This is because during livy session register method, exception 
> "IllegalArgumentException" is thrown without stopping the session:
> {code:java}
> def register(session: S): S = {
>     info(s"Registering new session ${session.id}")
>     synchronized {
>   session.name.foreach { sessionName =>
>     if (sessionsByName.contains(sessionName)) {
>   throw new IllegalArgumentException(s"Duplicate session name: 
> ${session.name}")
>     } else {
>   sessionsByName.put(sessionName, session)
>     }
>   }
>   sessions.put(session.id, session)
>   session.start()
>     }
>     session
>   }{code}
>  
> Reproduction scripts:
> curl -s -k -u username:password -X POST --data '\{"name": "duplicatedname", 
> "kind": "pyspark"}' -H "Content-Type: application/json" 
> 'https://myserver/livy/v1/sessions'



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (LIVY-641) Travis failed on "should end with status dead when batch session exits with no 0 return code"

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916318#comment-16916318
 ] 

Saisai Shao edited comment on LIVY-641 at 8/27/19 3:58 AM:
---

Issue resolved by pull request 214
[https://github.com/apache/incubator-livy/pull/214|https://github.com/apache/incubator-livy/pull/214]


was (Author: jerryshao):
Issue resolved by pull request 214
[https://github.com/apache/incubator-livy/pull/214|https://github.com/apache/incubator-livy/pull/184]

> Travis failed on "should end with status dead when batch session exits with 
> no 0 return code"
> -
>
> Key: LIVY-641
> URL: https://issues.apache.org/jira/browse/LIVY-641
> Project: Livy
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.6.0
>Reporter: jiewang
>Assignee: jiewang
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> BatchSessionSpec:
>  2463A Batch process
>  2464- should create a process (2 seconds, 103 milliseconds)
>  2465- should update appId and appInfo (30 milliseconds)
>  {color:#FF}2466- should end with status dead when batch session exits 
> with no 0 return code *** FAILED *** (121 milliseconds){color}
> {color:#FF} 2467 false was not true (BatchSessionSpec.scala:138){color}
>  2468- should recover session (name = None) (17 milliseconds)
>  2469- should recover session (name = Some(Test Batch Session)) (4 
> milliseconds)
>  2470- should recover session (name = null) (15 milliseconds)
>  please reference to 
> [https://travis-ci.org/apache/incubator-livy/builds/572562376?utm_source=github_status&utm_medium=notification]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (LIVY-641) Travis failed on "should end with status dead when batch session exits with no 0 return code"

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-641.
--
Fix Version/s: 0.7.0
 Assignee: jiewang
   Resolution: Fixed

Issue resolved by pull request 214
[https://github.com/apache/incubator-livy/pull/214|https://github.com/apache/incubator-livy/pull/184]

> Travis failed on "should end with status dead when batch session exits with 
> no 0 return code"
> -
>
> Key: LIVY-641
> URL: https://issues.apache.org/jira/browse/LIVY-641
> Project: Livy
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.6.0
>Reporter: jiewang
>Assignee: jiewang
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> BatchSessionSpec:
>  2463A Batch process
>  2464- should create a process (2 seconds, 103 milliseconds)
>  2465- should update appId and appInfo (30 milliseconds)
>  {color:#FF}2466- should end with status dead when batch session exits 
> with no 0 return code *** FAILED *** (121 milliseconds){color}
> {color:#FF} 2467 false was not true (BatchSessionSpec.scala:138){color}
>  2468- should recover session (name = None) (17 milliseconds)
>  2469- should recover session (name = Some(Test Batch Session)) (4 
> milliseconds)
>  2470- should recover session (name = null) (15 milliseconds)
>  please reference to 
> [https://travis-ci.org/apache/incubator-livy/builds/572562376?utm_source=github_status&utm_medium=notification]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (LIVY-639) Is it possible to add start time and completion time and duration to the statements web ui interface?

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916306#comment-16916306
 ] 

Saisai Shao commented on LIVY-639:
--

Issue resolved by pull request 201
https://github.com/apache/incubator-livy/pull/201

> Is it possible to add start time and completion time and duration to the 
> statements web ui interface?
> -
>
> Key: LIVY-639
> URL: https://issues.apache.org/jira/browse/LIVY-639
> Project: Livy
>  Issue Type: Improvement
>  Components: REPL, RSC, Server
>Affects Versions: 0.6.0
>Reporter: zhang peng
>Assignee: zhang peng
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: DeepinScreenshot_select-area_20190815153353.png
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Many times I need to know the execution time and start time of the statement, 
> but now the livy web ui does not support this. I have improved livy and have 
> written the code, I hope to get community support and submit PR.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Assigned] (LIVY-639) Is it possible to add start time and completion time and duration to the statements web ui interface?

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao reassigned LIVY-639:


Assignee: zhang peng

> Is it possible to add start time and completion time and duration to the 
> statements web ui interface?
> -
>
> Key: LIVY-639
> URL: https://issues.apache.org/jira/browse/LIVY-639
> Project: Livy
>  Issue Type: Improvement
>  Components: REPL, RSC, Server
>Affects Versions: 0.6.0
>Reporter: zhang peng
>Assignee: zhang peng
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: DeepinScreenshot_select-area_20190815153353.png
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Many times I need to know the execution time and start time of the statement, 
> but now the livy web ui does not support this. I have improved livy and have 
> written the code, I hope to get community support and submit PR.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (LIVY-639) Is it possible to add start time and completion time and duration to the statements web ui interface?

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-639.
--
Fix Version/s: 0.7.0
   Resolution: Fixed

> Is it possible to add start time and completion time and duration to the 
> statements web ui interface?
> -
>
> Key: LIVY-639
> URL: https://issues.apache.org/jira/browse/LIVY-639
> Project: Livy
>  Issue Type: Improvement
>  Components: REPL, RSC, Server
>Affects Versions: 0.6.0
>Reporter: zhang peng
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: DeepinScreenshot_select-area_20190815153353.png
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Many times I need to know the execution time and start time of the statement, 
> but now the livy web ui does not support this. I have improved livy and have 
> written the code, I hope to get community support and submit PR.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Assigned] (LIVY-648) Wrong return message in cancel statement documentation

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao reassigned LIVY-648:


Assignee: Oleksandr Shevchenko

> Wrong return message in cancel statement documentation
> --
>
> Key: LIVY-648
> URL: https://issues.apache.org/jira/browse/LIVY-648
> Project: Livy
>  Issue Type: Documentation
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Trivial
> Fix For: 0.7.0
>
> Attachments: image-2019-08-24-00-54-29-000.png, 
> image-2019-08-24-00-58-11-847.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> A trivial mistake in the documentation. "Canceled" vs "cancelled" (two Ls).
> [https://github.com/apache/incubator-livy/blob/80daadef02ae57b2a5487c6f92e0f7df558d4864/docs/rest-api.md#L308]
> !image-2019-08-24-00-54-29-000.png|width=1529,height=278!
> !image-2019-08-24-00-58-11-847.png!
> We can just fix documentation or unify all names across the project.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (LIVY-648) Wrong return message in cancel statement documentation

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-648.
--
Fix Version/s: 0.7.0
   Resolution: Fixed

Issue resolved by pull request 210
[https://github.com/apache/incubator-livy/pull/210]

> Wrong return message in cancel statement documentation
> --
>
> Key: LIVY-648
> URL: https://issues.apache.org/jira/browse/LIVY-648
> Project: Livy
>  Issue Type: Documentation
>Reporter: Oleksandr Shevchenko
>Priority: Trivial
> Fix For: 0.7.0
>
> Attachments: image-2019-08-24-00-54-29-000.png, 
> image-2019-08-24-00-58-11-847.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> A trivial mistake in the documentation. "Canceled" vs "cancelled" (two Ls).
> [https://github.com/apache/incubator-livy/blob/80daadef02ae57b2a5487c6f92e0f7df558d4864/docs/rest-api.md#L308]
> !image-2019-08-24-00-54-29-000.png|width=1529,height=278!
> !image-2019-08-24-00-58-11-847.png!
> We can just fix documentation or unify all names across the project.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated SPARK-28849:

Description: 
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until it is killed manually. Here's 
the log you can see, there's no any log after spilling the shuffle data to 
disk, but the executor is still alive.

 !95330.png! 

And here is the thread dump, we could see that it always calls native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system call, we found that this thread is always 
calling {{fstat}}, and the system usage is pretty high, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.

  was:
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until it is killed manually. Here's 
the log you can see, there's no any log after spilling the shuffle data to 
disk, but the executor is still alive.

 !95330.png! 

And here is the thread dump, we could see that it always calls native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, and the system usage is pretty high, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.


> Spark's UnsafeShuffleWriter may run into infinite loop in transferTo 
> occasionally
> -
>
> Key: SPARK-28849
> URL: https://issues.apache.org/jira/browse/SPARK-28849
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.1
>Reporter: Saisai Shao
>Priority: Major
> Attachments: 91ADA.png, 95330.png, D18F4.png
>
>
> Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
> {{transferTo}} occasionally. What we saw is that when merging shuffle temp 
> file, the task is hung for several hours until it is killed manually. Here's 
> the log you can see, there's no any log after spilling the shuffle data to 
> disk, but the executor is still alive.
>  !95330.png! 
> And here is the thread dump, we could see that it always calls native method 
> {{size0}}.
>  !91ADA.png! 
> And we use strace to trace the system call, we found that this thread is 
> always calling {{fstat}}, and the system usage is pretty high, here is the 
> screenshot. 
>  !D18F4.png! 
> We didn't find the root cause here, I guess it might be related to FS or disk 
> issue. Anyway we should figure out a way to fail fast in a such scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated SPARK-28849:

Description: 
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until it is killed manually. Here's 
the log you can see, there's no any log after spilling the shuffle data to 
disk, but the executor is still alive.

 !95330.png! 

And here is the thread dump, we could see that it always calls native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, and the system usage is pretty high, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.

  was:
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until it is killed manually. Here's 
the log you can see, there's no any log after spill the shuffle data to disk.

 !95330.png! 

And here is the thread dump, we could see that it always calls native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, and the system usage is pretty high, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.


> Spark's UnsafeShuffleWriter may run into infinite loop in transferTo 
> occasionally
> -
>
> Key: SPARK-28849
> URL: https://issues.apache.org/jira/browse/SPARK-28849
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.1
>Reporter: Saisai Shao
>Priority: Major
> Attachments: 91ADA.png, 95330.png, D18F4.png
>
>
> Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
> {{transferTo}} occasionally. What we saw is that when merging shuffle temp 
> file, the task is hung for several hours until it is killed manually. Here's 
> the log you can see, there's no any log after spilling the shuffle data to 
> disk, but the executor is still alive.
>  !95330.png! 
> And here is the thread dump, we could see that it always calls native method 
> {{size0}}.
>  !91ADA.png! 
> And we use strace to trace the system, we found that this thread is always 
> calling {{fstat}}, and the system usage is pretty high, here is the 
> screenshot. 
>  !D18F4.png! 
> We didn't find the root cause here, I guess it might be related to FS or disk 
> issue. Anyway we should figure out a way to fail fast in a such scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated SPARK-28849:

Description: 
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until it is killed manually. Here's 
the log you can see, there's no any log after spill the shuffle data to disk.

 !95330.png! 

And here is the thread dump, we could see that it always calls native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, and the system usage is pretty high, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.

  was:
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until killed manually. Here's the log 
you can see, there's no any log after spill the shuffle files to disk.

 !95330.png! 

And here is the thread dump, we could see that it always calls native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, and the system usage is pretty high, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.


> Spark's UnsafeShuffleWriter may run into infinite loop in transferTo 
> occasionally
> -
>
> Key: SPARK-28849
> URL: https://issues.apache.org/jira/browse/SPARK-28849
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.1
>Reporter: Saisai Shao
>Priority: Major
> Attachments: 91ADA.png, 95330.png, D18F4.png
>
>
> Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
> {{transferTo}} occasionally. What we saw is that when merging shuffle temp 
> file, the task is hung for several hours until it is killed manually. Here's 
> the log you can see, there's no any log after spill the shuffle data to disk.
>  !95330.png! 
> And here is the thread dump, we could see that it always calls native method 
> {{size0}}.
>  !91ADA.png! 
> And we use strace to trace the system, we found that this thread is always 
> calling {{fstat}}, and the system usage is pretty high, here is the 
> screenshot. 
>  !D18F4.png! 
> We didn't find the root cause here, I guess it might be related to FS or disk 
> issue. Anyway we should figure out a way to fail fast in a such scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated SPARK-28849:

Description: 
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until killed manually. Here's the log 
you can see, there's no any log after spill the shuffle files to disk.

 !95330.png! 

And here is the thread dump, we could see that it always calls native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.

  was:
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until killed manually. Here's the log 
you can see, there's no any log after spill the shuffle files to disk.

 !95330.png! 

And here is the thread dump, we could see that it is calling native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.


> Spark's UnsafeShuffleWriter may run into infinite loop in transferTo 
> occasionally
> -
>
> Key: SPARK-28849
> URL: https://issues.apache.org/jira/browse/SPARK-28849
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.1
>Reporter: Saisai Shao
>Priority: Major
> Attachments: 91ADA.png, 95330.png, D18F4.png
>
>
> Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
> {{transferTo}} occasionally. What we saw is that when merging shuffle temp 
> file, the task is hung for several hours until killed manually. Here's the 
> log you can see, there's no any log after spill the shuffle files to disk.
>  !95330.png! 
> And here is the thread dump, we could see that it always calls native method 
> {{size0}}.
>  !91ADA.png! 
> And we use strace to trace the system, we found that this thread is always 
> calling {{fstat}}, here is the screenshot. 
>  !D18F4.png! 
> We didn't find the root cause here, I guess it might be related to FS or disk 
> issue. Anyway we should figure out a way to fail fast in a such scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated SPARK-28849:

Description: 
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until killed manually. Here's the log 
you can see, there's no any log after spill the shuffle files to disk.

 !95330.png! 

And here is the thread dump, we could see that it always calls native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, and the system usage is pretty high, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.

  was:
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until killed manually. Here's the log 
you can see, there's no any log after spill the shuffle files to disk.

 !95330.png! 

And here is the thread dump, we could see that it always calls native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.


> Spark's UnsafeShuffleWriter may run into infinite loop in transferTo 
> occasionally
> -
>
> Key: SPARK-28849
> URL: https://issues.apache.org/jira/browse/SPARK-28849
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.1
>Reporter: Saisai Shao
>Priority: Major
> Attachments: 91ADA.png, 95330.png, D18F4.png
>
>
> Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
> {{transferTo}} occasionally. What we saw is that when merging shuffle temp 
> file, the task is hung for several hours until killed manually. Here's the 
> log you can see, there's no any log after spill the shuffle files to disk.
>  !95330.png! 
> And here is the thread dump, we could see that it always calls native method 
> {{size0}}.
>  !91ADA.png! 
> And we use strace to trace the system, we found that this thread is always 
> calling {{fstat}}, and the system usage is pretty high, here is the 
> screenshot. 
>  !D18F4.png! 
> We didn't find the root cause here, I guess it might be related to FS or disk 
> issue. Anyway we should figure out a way to fail fast in a such scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated SPARK-28849:

Description: 
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until killed manually. Here's the log 
you can see, there's no any log after spill the shuffle files to disk.

 !95330.png! 

And here is the thread dump, we could see that it is calling native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.

  was:
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until killed manually. Here's the log 
you can see, there's no any log after spill the shuffle files to disk for 
several hours.

 !95330.png! 

And here is the thread dump, we could see that it is calling native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.


> Spark's UnsafeShuffleWriter may run into infinite loop in transferTo 
> occasionally
> -
>
> Key: SPARK-28849
> URL: https://issues.apache.org/jira/browse/SPARK-28849
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.1
>Reporter: Saisai Shao
>Priority: Major
> Attachments: 91ADA.png, 95330.png, D18F4.png
>
>
> Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
> {{transferTo}} occasionally. What we saw is that when merging shuffle temp 
> file, the task is hung for several hours until killed manually. Here's the 
> log you can see, there's no any log after spill the shuffle files to disk.
>  !95330.png! 
> And here is the thread dump, we could see that it is calling native method 
> {{size0}}.
>  !91ADA.png! 
> And we use strace to trace the system, we found that this thread is always 
> calling {{fstat}}, here is the screenshot. 
>  !D18F4.png! 
> We didn't find the root cause here, I guess it might be related to FS or disk 
> issue. Anyway we should figure out a way to fail fast in a such scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated SPARK-28849:

Attachment: D18F4.png
95330.png
91ADA.png

> Spark's UnsafeShuffleWriter may run into infinite loop in transferTo 
> occasionally
> -
>
> Key: SPARK-28849
> URL: https://issues.apache.org/jira/browse/SPARK-28849
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.1
>Reporter: Saisai Shao
>Priority: Major
> Attachments: 91ADA.png, 95330.png, D18F4.png
>
>
> Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
> {{transferTo}} occasionally. What we saw is that when merging shuffle temp 
> file, the task is hung for several hours until killed manually. Here's the 
> log you can see, there's no any log after spill the shuffle files to disk for 
> several hours.
> And here is the thread dump, we could see that it is calling native method 
> {{size0}}.
> And we use strace to trace the system, we found that this thread is always 
> calling {{fstat}}, here is the screenshot. 
> We didn't find the root cause here, I guess it might be related to FS or disk 
> issue. Anyway we should figure out a way to fail fast in a such scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated SPARK-28849:

Description: 
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until killed manually. Here's the log 
you can see, there's no any log after spill the shuffle files to disk for 
several hours.

 !95330.png! 

And here is the thread dump, we could see that it is calling native method 
{{size0}}.

 !91ADA.png! 

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, here is the screenshot. 

 !D18F4.png! 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.

  was:
Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until killed manually. Here's the log 
you can see, there's no any log after spill the shuffle files to disk for 
several hours.

And here is the thread dump, we could see that it is calling native method 
{{size0}}.

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, here is the screenshot. 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.


> Spark's UnsafeShuffleWriter may run into infinite loop in transferTo 
> occasionally
> -
>
> Key: SPARK-28849
> URL: https://issues.apache.org/jira/browse/SPARK-28849
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.1
>Reporter: Saisai Shao
>Priority: Major
> Attachments: 91ADA.png, 95330.png, D18F4.png
>
>
> Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
> {{transferTo}} occasionally. What we saw is that when merging shuffle temp 
> file, the task is hung for several hours until killed manually. Here's the 
> log you can see, there's no any log after spill the shuffle files to disk for 
> several hours.
>  !95330.png! 
> And here is the thread dump, we could see that it is calling native method 
> {{size0}}.
>  !91ADA.png! 
> And we use strace to trace the system, we found that this thread is always 
> calling {{fstat}}, here is the screenshot. 
>  !D18F4.png! 
> We didn't find the root cause here, I guess it might be related to FS or disk 
> issue. Anyway we should figure out a way to fail fast in a such scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

[Saisai Shao (Jira)]

Saisai Shao created SPARK-28849:
---

 Summary: Spark's UnsafeShuffleWriter may run into infinite loop in 
transferTo occasionally
 Key: SPARK-28849
 URL: https://issues.apache.org/jira/browse/SPARK-28849
 Project: Spark
  Issue Type: Bug
  Components: Spark Core
Affects Versions: 2.3.1
Reporter: Saisai Shao


Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling 
{{transferTo}} occasionally. What we saw is that when merging shuffle temp 
file, the task is hung for several hours until killed manually. Here's the log 
you can see, there's no any log after spill the shuffle files to disk for 
several hours.

And here is the thread dump, we could see that it is calling native method 
{{size0}}.

And we use strace to trace the system, we found that this thread is always 
calling {{fstat}}, here is the screenshot. 

We didn't find the root cause here, I guess it might be related to FS or disk 
issue. Anyway we should figure out a way to fail fast in a such scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (LIVY-637) get NullPointerException when create database using thriftserver

[Saisai Shao (Jira)]

 [ 
https://issues.apache.org/jira/browse/LIVY-637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao resolved LIVY-637.
--
Fix Version/s: 0.7.0
   Resolution: Fixed

> get NullPointerException when create database using thriftserver
> 
>
> Key: LIVY-637
> URL: https://issues.apache.org/jira/browse/LIVY-637
> Project: Livy
>  Issue Type: Bug
>  Components: Thriftserver
>Affects Versions: 0.6.0
>Reporter: mingchao zhao
>Assignee: mingchao zhao
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: create.png, drop.png, use.png
>
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> When I connected thriftserver with spark beeline. NullPointerException occurs 
> when  execute the following SQL. This exception does not affect the final 
> execution result.
> create database test;
> use test;
> drop database test;
> 0: jdbc:hive2://localhost:10090> create database test;
>  java.lang.NullPointerException
>  at org.apache.hive.service.cli.ColumnBasedSet.(ColumnBasedSet.java:50)
>  at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:37)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:368)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>  Error: Error retrieving next row (state=,code=0)
>  0: jdbc:hive2://localhost:10090> use test;
>  java.lang.NullPointerException
>  at org.apache.hive.service.cli.ColumnBasedSet.(ColumnBasedSet.java:50)
>  at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:37)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:368)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>  Error: Error retrieving next row (state=,code=0)
> 0: jdbc:hive2://localhost:10090> drop database test;
>  java.lang.NullPointerException
>  at org.apache.hive.service.cli.ColumnBasedSet.(ColumnBasedSet.java:50)
>  at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:37)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:368)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>  Error: Error retrieving next row (state=,code=0)
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (LIVY-637) get NullPointerException when create database using thriftserver

[Saisai Shao (Jira)]

[ 
https://issues.apache.org/jira/browse/LIVY-637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913001#comment-16913001
 ] 

Saisai Shao commented on LIVY-637:
--

Issue resolved by pull request 200
https://github.com/apache/incubator-livy/pull/200

> get NullPointerException when create database using thriftserver
> 
>
> Key: LIVY-637
> URL: https://issues.apache.org/jira/browse/LIVY-637
> Project: Livy
>  Issue Type: Bug
>  Components: Thriftserver
>Affects Versions: 0.6.0
>Reporter: mingchao zhao
>Assignee: mingchao zhao
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: create.png, drop.png, use.png
>
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> When I connected thriftserver with spark beeline. NullPointerException occurs 
> when  execute the following SQL. This exception does not affect the final 
> execution result.
> create database test;
> use test;
> drop database test;
> 0: jdbc:hive2://localhost:10090> create database test;
>  java.lang.NullPointerException
>  at org.apache.hive.service.cli.ColumnBasedSet.(ColumnBasedSet.java:50)
>  at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:37)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:368)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>  Error: Error retrieving next row (state=,code=0)
>  0: jdbc:hive2://localhost:10090> use test;
>  java.lang.NullPointerException
>  at org.apache.hive.service.cli.ColumnBasedSet.(ColumnBasedSet.java:50)
>  at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:37)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:368)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>  Error: Error retrieving next row (state=,code=0)
> 0: jdbc:hive2://localhost:10090> drop database test;
>  java.lang.NullPointerException
>  at org.apache.hive.service.cli.ColumnBasedSet.(ColumnBasedSet.java:50)
>  at org.apache.hive.service.cli.RowSetFactory.create(RowSetFactory.java:37)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:368)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>  Error: Error retrieving next row (state=,code=0)
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)