Re: [discuss] SparkR CRAN feasibility check server problem
Just got reply from CRAN admin. It should be fixed now. Hyukjin Kwon wrote > Thanks, Liang-chi. > > On Thu, 13 Dec 2018, 8:29 am Liang-Chi Hsieh < > viirya@ > wrote: > > > >> Sorry for late. There is a malformed record at CRAN package page again. >> I've >> already asked CRAN admin for help. It should be fixed soon according to >> past >> experience. >> >> Related discussion will be in >> https://issues.apache.org/jira/browse/SPARK-24152. I will post here if I >> get >> reply from CRAN admin. >> >> Thanks. >> >> >> Liang-Chi Hsieh wrote >> > Thanks for letting me know! I will look into it and ask CRAN admin for >> > help. >> > >> > >> > Hyukjin Kwon wrote >> >> Looks it's happening again. Liang-Chi, do you mind if I ask it again? >> >> >> >> FYI, R 3.4 is officially deprecated as of >> >> https://github.com/apache/spark/pull/23012 >> >> We could upgrade R version to 3.4.x in Jenkins, which deals with the >> >> malformed(?) responses after 3.0 release. >> >> Then, we could get rid of this problem..! >> >> >> >> 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon < >> > >> >> gurwls223@ >> > >> >> >님이 작성: >> >> >> >>> I made a PR to officially drop R prior to version 3.4 ( >> >>> https://github.com/apache/spark/pull/23012). >> >>> The tests will probably fail for now since it produces warnings for >> >>> using >> >>> R 3.1.x. >> >>> >> >>> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung < >> > >> >> felixcheung_m@ >> > >> >> >님이 작성: >> >>> >> >>>> It’s a great point about min R version. From what I see, mostly >> because >> >>>> of fixes and packages support, most users of R are fairly up to >> date? >> >>>> So >> >>>> perhaps 3.4 as min version is reasonable esp. for Spark 3. >> >>>> >> >>>> Are we getting traction with CRAN sysadmin? It seems like this has >> been >> >>>> broken a few times. >> >>>> >> >>>> >> >>>> -- >> >>>> *From:* Liang-Chi Hsieh < >> > >> >> viirya@ >> > >> >> > >> >>>> *Sent:* Saturday, November 10, 2018 2:32 AM >> >>>> *To:* >> > >> >> dev@.apache >> > >> >>>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server >> problem >> >>>> >> >>>> >> >>>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion. >> >>>> >> >>>> I don't know how higher versions of R are widely used across R >> >>>> community. >> >>>> If >> >>>> R version 3.1.x was not very commonly used, I think we can discuss >> to >> >>>> upgrade minimum R version in next Spark version. >> >>>> >> >>>> If we ended up with not upgrading, we can discuss with CRAN sysadmin >> to >> >>>> fix >> >>>> it by the service side automatically that prevents malformed R >> packages >> >>>> info. So we don't need to fix it manually every time. >> >>>> >> >>>> >> >>>> >> >>>> Hyukjin Kwon wrote >> >>>> >> Can upgrading R able to fix the issue. Is this perhaps not >> >>>> necessarily >> >>>> > malform but some new format for new versions perhaps? >> >>>> > That's my guess. I am not totally sure about it tho. >> >>>> > >> >>>> >> Anyway we should consider upgrading R version if that fixes the >> >>>> problem. >> >>>> > Yea, we should. If we should, it should be more them R 3.4. Maybe >> >>>> it's >> >>>> > good >> >>>> > time to start to talk about minimum R version. 3.1.x is too old. >> It's >> >>>> > released 4.5 years ago. >> >>>> > R 3.4.0 is released 1.5 years ago. Considering the timing for >> Spark >> >>>> 3.0, >> >>>> > deprecating lower versions, bumping up R to 3.4 might be
Re: [discuss] SparkR CRAN feasibility check server problem
Thanks, Liang-chi. On Thu, 13 Dec 2018, 8:29 am Liang-Chi Hsieh > Sorry for late. There is a malformed record at CRAN package page again. > I've > already asked CRAN admin for help. It should be fixed soon according to > past > experience. > > Related discussion will be in > https://issues.apache.org/jira/browse/SPARK-24152. I will post here if I > get > reply from CRAN admin. > > Thanks. > > > Liang-Chi Hsieh wrote > > Thanks for letting me know! I will look into it and ask CRAN admin for > > help. > > > > > > Hyukjin Kwon wrote > >> Looks it's happening again. Liang-Chi, do you mind if I ask it again? > >> > >> FYI, R 3.4 is officially deprecated as of > >> https://github.com/apache/spark/pull/23012 > >> We could upgrade R version to 3.4.x in Jenkins, which deals with the > >> malformed(?) responses after 3.0 release. > >> Then, we could get rid of this problem..! > >> > >> 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon < > > > >> gurwls223@ > > > >> >님이 작성: > >> > >>> I made a PR to officially drop R prior to version 3.4 ( > >>> https://github.com/apache/spark/pull/23012). > >>> The tests will probably fail for now since it produces warnings for > >>> using > >>> R 3.1.x. > >>> > >>> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung < > > > >> felixcheung_m@ > > > >> >님이 작성: > >>> > >>>> It’s a great point about min R version. From what I see, mostly > because > >>>> of fixes and packages support, most users of R are fairly up to date? > >>>> So > >>>> perhaps 3.4 as min version is reasonable esp. for Spark 3. > >>>> > >>>> Are we getting traction with CRAN sysadmin? It seems like this has > been > >>>> broken a few times. > >>>> > >>>> > >>>> -- > >>>> *From:* Liang-Chi Hsieh < > > > >> viirya@ > > > >> > > >>>> *Sent:* Saturday, November 10, 2018 2:32 AM > >>>> *To:* > > > >> dev@.apache > > > >>>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem > >>>> > >>>> > >>>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion. > >>>> > >>>> I don't know how higher versions of R are widely used across R > >>>> community. > >>>> If > >>>> R version 3.1.x was not very commonly used, I think we can discuss to > >>>> upgrade minimum R version in next Spark version. > >>>> > >>>> If we ended up with not upgrading, we can discuss with CRAN sysadmin > to > >>>> fix > >>>> it by the service side automatically that prevents malformed R > packages > >>>> info. So we don't need to fix it manually every time. > >>>> > >>>> > >>>> > >>>> Hyukjin Kwon wrote > >>>> >> Can upgrading R able to fix the issue. Is this perhaps not > >>>> necessarily > >>>> > malform but some new format for new versions perhaps? > >>>> > That's my guess. I am not totally sure about it tho. > >>>> > > >>>> >> Anyway we should consider upgrading R version if that fixes the > >>>> problem. > >>>> > Yea, we should. If we should, it should be more them R 3.4. Maybe > >>>> it's > >>>> > good > >>>> > time to start to talk about minimum R version. 3.1.x is too old. > It's > >>>> > released 4.5 years ago. > >>>> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark > >>>> 3.0, > >>>> > deprecating lower versions, bumping up R to 3.4 might be reasonable > >>>> > option. > >>>> > > >>>> > Adding Shane as well. > >>>> > > >>>> > If we ended up with not upgrading it, I will forward this email to > >>>> CRAN > >>>> > sysadmin to discuss further anyway. > >>>> > > >>>> > > >>>> > > >>>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung < > >>>> > >>>> > felixcheung@ > >>>> > >>>> > >
Re: [discuss] SparkR CRAN feasibility check server problem
Sorry for late. There is a malformed record at CRAN package page again. I've already asked CRAN admin for help. It should be fixed soon according to past experience. Related discussion will be in https://issues.apache.org/jira/browse/SPARK-24152. I will post here if I get reply from CRAN admin. Thanks. Liang-Chi Hsieh wrote > Thanks for letting me know! I will look into it and ask CRAN admin for > help. > > > Hyukjin Kwon wrote >> Looks it's happening again. Liang-Chi, do you mind if I ask it again? >> >> FYI, R 3.4 is officially deprecated as of >> https://github.com/apache/spark/pull/23012 >> We could upgrade R version to 3.4.x in Jenkins, which deals with the >> malformed(?) responses after 3.0 release. >> Then, we could get rid of this problem..! >> >> 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon < > >> gurwls223@ > >> >님이 작성: >> >>> I made a PR to officially drop R prior to version 3.4 ( >>> https://github.com/apache/spark/pull/23012). >>> The tests will probably fail for now since it produces warnings for >>> using >>> R 3.1.x. >>> >>> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung < > >> felixcheung_m@ > >> >님이 작성: >>> >>>> It’s a great point about min R version. From what I see, mostly because >>>> of fixes and packages support, most users of R are fairly up to date? >>>> So >>>> perhaps 3.4 as min version is reasonable esp. for Spark 3. >>>> >>>> Are we getting traction with CRAN sysadmin? It seems like this has been >>>> broken a few times. >>>> >>>> >>>> -- >>>> *From:* Liang-Chi Hsieh < > >> viirya@ > >> > >>>> *Sent:* Saturday, November 10, 2018 2:32 AM >>>> *To:* > >> dev@.apache > >>>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem >>>> >>>> >>>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion. >>>> >>>> I don't know how higher versions of R are widely used across R >>>> community. >>>> If >>>> R version 3.1.x was not very commonly used, I think we can discuss to >>>> upgrade minimum R version in next Spark version. >>>> >>>> If we ended up with not upgrading, we can discuss with CRAN sysadmin to >>>> fix >>>> it by the service side automatically that prevents malformed R packages >>>> info. So we don't need to fix it manually every time. >>>> >>>> >>>> >>>> Hyukjin Kwon wrote >>>> >> Can upgrading R able to fix the issue. Is this perhaps not >>>> necessarily >>>> > malform but some new format for new versions perhaps? >>>> > That's my guess. I am not totally sure about it tho. >>>> > >>>> >> Anyway we should consider upgrading R version if that fixes the >>>> problem. >>>> > Yea, we should. If we should, it should be more them R 3.4. Maybe >>>> it's >>>> > good >>>> > time to start to talk about minimum R version. 3.1.x is too old. It's >>>> > released 4.5 years ago. >>>> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark >>>> 3.0, >>>> > deprecating lower versions, bumping up R to 3.4 might be reasonable >>>> > option. >>>> > >>>> > Adding Shane as well. >>>> > >>>> > If we ended up with not upgrading it, I will forward this email to >>>> CRAN >>>> > sysadmin to discuss further anyway. >>>> > >>>> > >>>> > >>>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung < >>>> >>>> > felixcheung@ >>>> >>>> > >님이 작성: >>>> > >>>> >> Thanks for being this up and much appreciate with keeping on top of >>>> this >>>> >> at all times. >>>> >> >>>> >> Can upgrading R able to fix the issue. Is this perhaps not >>>> necessarily >>>> >> malform but some new format for new versions perhaps? Anyway we >>>> should >>>> >> consider upgrading R version if that fixes the problem. >>>> >> >>>> >> As an option we could also disable the repo check in Jenkins but I >&
Re: [discuss] SparkR CRAN feasibility check server problem
Thanks for letting me know! I will look into it and ask CRAN admin for help. Hyukjin Kwon wrote > Looks it's happening again. Liang-Chi, do you mind if I ask it again? > > FYI, R 3.4 is officially deprecated as of > https://github.com/apache/spark/pull/23012 > We could upgrade R version to 3.4.x in Jenkins, which deals with the > malformed(?) responses after 3.0 release. > Then, we could get rid of this problem..! > > 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon < > gurwls223@ > >님이 작성: > >> I made a PR to officially drop R prior to version 3.4 ( >> https://github.com/apache/spark/pull/23012). >> The tests will probably fail for now since it produces warnings for using >> R 3.1.x. >> >> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung < > felixcheung_m@ > >님이 작성: >> >>> It’s a great point about min R version. From what I see, mostly because >>> of fixes and packages support, most users of R are fairly up to date? So >>> perhaps 3.4 as min version is reasonable esp. for Spark 3. >>> >>> Are we getting traction with CRAN sysadmin? It seems like this has been >>> broken a few times. >>> >>> >>> ---------- >>> *From:* Liang-Chi Hsieh < > viirya@ > > >>> *Sent:* Saturday, November 10, 2018 2:32 AM >>> *To:* > dev@.apache >>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem >>> >>> >>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion. >>> >>> I don't know how higher versions of R are widely used across R >>> community. >>> If >>> R version 3.1.x was not very commonly used, I think we can discuss to >>> upgrade minimum R version in next Spark version. >>> >>> If we ended up with not upgrading, we can discuss with CRAN sysadmin to >>> fix >>> it by the service side automatically that prevents malformed R packages >>> info. So we don't need to fix it manually every time. >>> >>> >>> >>> Hyukjin Kwon wrote >>> >> Can upgrading R able to fix the issue. Is this perhaps not >>> necessarily >>> > malform but some new format for new versions perhaps? >>> > That's my guess. I am not totally sure about it tho. >>> > >>> >> Anyway we should consider upgrading R version if that fixes the >>> problem. >>> > Yea, we should. If we should, it should be more them R 3.4. Maybe it's >>> > good >>> > time to start to talk about minimum R version. 3.1.x is too old. It's >>> > released 4.5 years ago. >>> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark >>> 3.0, >>> > deprecating lower versions, bumping up R to 3.4 might be reasonable >>> > option. >>> > >>> > Adding Shane as well. >>> > >>> > If we ended up with not upgrading it, I will forward this email to >>> CRAN >>> > sysadmin to discuss further anyway. >>> > >>> > >>> > >>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung < >>> >>> > felixcheung@ >>> >>> > >님이 작성: >>> > >>> >> Thanks for being this up and much appreciate with keeping on top of >>> this >>> >> at all times. >>> >> >>> >> Can upgrading R able to fix the issue. Is this perhaps not >>> necessarily >>> >> malform but some new format for new versions perhaps? Anyway we >>> should >>> >> consider upgrading R version if that fixes the problem. >>> >> >>> >> As an option we could also disable the repo check in Jenkins but I >>> can >>> >> see >>> >> that could also be problematic. >>> >> >>> >> >>> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon < >>> >>> > gurwls223@ >>> >>> > > wrote: >>> >> >>> >>> Hi all, >>> >>> >>> >>> I want to raise the CRAN failure issue because it started to block >>> Spark >>> >>> PRs time to time. Since the number >>> >>> of PRs grows hugely in Spark community, this is critical to not >>> block >>> >>> other PRs. >>> >>> >>> >>> There has been a problem at CRAN (See >>> >>> https://github.com/ap
Re: [discuss] SparkR CRAN feasibility check server problem
Looks it's happening again. Liang-Chi, do you mind if I ask it again? FYI, R 3.4 is officially deprecated as of https://github.com/apache/spark/pull/23012 We could upgrade R version to 3.4.x in Jenkins, which deals with the malformed(?) responses after 3.0 release. Then, we could get rid of this problem..! 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon 님이 작성: > I made a PR to officially drop R prior to version 3.4 ( > https://github.com/apache/spark/pull/23012). > The tests will probably fail for now since it produces warnings for using > R 3.1.x. > > 2018년 11월 11일 (일) 오전 3:00, Felix Cheung 님이 작성: > >> It’s a great point about min R version. From what I see, mostly because >> of fixes and packages support, most users of R are fairly up to date? So >> perhaps 3.4 as min version is reasonable esp. for Spark 3. >> >> Are we getting traction with CRAN sysadmin? It seems like this has been >> broken a few times. >> >> >> -- >> *From:* Liang-Chi Hsieh >> *Sent:* Saturday, November 10, 2018 2:32 AM >> *To:* dev@spark.apache.org >> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem >> >> >> Yeah, thanks Hyukjin Kwon for bringing this up for discussion. >> >> I don't know how higher versions of R are widely used across R community. >> If >> R version 3.1.x was not very commonly used, I think we can discuss to >> upgrade minimum R version in next Spark version. >> >> If we ended up with not upgrading, we can discuss with CRAN sysadmin to >> fix >> it by the service side automatically that prevents malformed R packages >> info. So we don't need to fix it manually every time. >> >> >> >> Hyukjin Kwon wrote >> >> Can upgrading R able to fix the issue. Is this perhaps not necessarily >> > malform but some new format for new versions perhaps? >> > That's my guess. I am not totally sure about it tho. >> > >> >> Anyway we should consider upgrading R version if that fixes the >> problem. >> > Yea, we should. If we should, it should be more them R 3.4. Maybe it's >> > good >> > time to start to talk about minimum R version. 3.1.x is too old. It's >> > released 4.5 years ago. >> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0, >> > deprecating lower versions, bumping up R to 3.4 might be reasonable >> > option. >> > >> > Adding Shane as well. >> > >> > If we ended up with not upgrading it, I will forward this email to CRAN >> > sysadmin to discuss further anyway. >> > >> > >> > >> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung < >> >> > felixcheung@ >> >> > >님이 작성: >> > >> >> Thanks for being this up and much appreciate with keeping on top of >> this >> >> at all times. >> >> >> >> Can upgrading R able to fix the issue. Is this perhaps not necessarily >> >> malform but some new format for new versions perhaps? Anyway we should >> >> consider upgrading R version if that fixes the problem. >> >> >> >> As an option we could also disable the repo check in Jenkins but I can >> >> see >> >> that could also be problematic. >> >> >> >> >> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon < >> >> > gurwls223@ >> >> > > wrote: >> >> >> >>> Hi all, >> >>> >> >>> I want to raise the CRAN failure issue because it started to block >> Spark >> >>> PRs time to time. Since the number >> >>> of PRs grows hugely in Spark community, this is critical to not block >> >>> other PRs. >> >>> >> >>> There has been a problem at CRAN (See >> >>> https://github.com/apache/spark/pull/20005 for analysis). >> >>> To cut it short, the root cause is malformed package info from >> >>> https://cran.r-project.org/src/contrib/PACKAGES >> >>> from server side, and this had to be fixed by requesting it to CRAN >> >>> sysaadmin's help. >> >>> >> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am >> >>> pretty sure it's the same issue >> >>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved >> 2 >> >>> times >> >>> https://issues.apache.org/jira/browse/SPARK-22812 &g
Re: [discuss] SparkR CRAN feasibility check server problem
I made a PR to officially drop R prior to version 3.4 ( https://github.com/apache/spark/pull/23012). The tests will probably fail for now since it produces warnings for using R 3.1.x. 2018년 11월 11일 (일) 오전 3:00, Felix Cheung 님이 작성: > It’s a great point about min R version. From what I see, mostly because of > fixes and packages support, most users of R are fairly up to date? So > perhaps 3.4 as min version is reasonable esp. for Spark 3. > > Are we getting traction with CRAN sysadmin? It seems like this has been > broken a few times. > > > -- > *From:* Liang-Chi Hsieh > *Sent:* Saturday, November 10, 2018 2:32 AM > *To:* dev@spark.apache.org > *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem > > > Yeah, thanks Hyukjin Kwon for bringing this up for discussion. > > I don't know how higher versions of R are widely used across R community. > If > R version 3.1.x was not very commonly used, I think we can discuss to > upgrade minimum R version in next Spark version. > > If we ended up with not upgrading, we can discuss with CRAN sysadmin to fix > it by the service side automatically that prevents malformed R packages > info. So we don't need to fix it manually every time. > > > > Hyukjin Kwon wrote > >> Can upgrading R able to fix the issue. Is this perhaps not necessarily > > malform but some new format for new versions perhaps? > > That's my guess. I am not totally sure about it tho. > > > >> Anyway we should consider upgrading R version if that fixes the problem. > > Yea, we should. If we should, it should be more them R 3.4. Maybe it's > > good > > time to start to talk about minimum R version. 3.1.x is too old. It's > > released 4.5 years ago. > > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0, > > deprecating lower versions, bumping up R to 3.4 might be reasonable > > option. > > > > Adding Shane as well. > > > > If we ended up with not upgrading it, I will forward this email to CRAN > > sysadmin to discuss further anyway. > > > > > > > > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung < > > > felixcheung@ > > > >님이 작성: > > > >> Thanks for being this up and much appreciate with keeping on top of this > >> at all times. > >> > >> Can upgrading R able to fix the issue. Is this perhaps not necessarily > >> malform but some new format for new versions perhaps? Anyway we should > >> consider upgrading R version if that fixes the problem. > >> > >> As an option we could also disable the repo check in Jenkins but I can > >> see > >> that could also be problematic. > >> > >> > >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon < > > > gurwls223@ > > > > wrote: > >> > >>> Hi all, > >>> > >>> I want to raise the CRAN failure issue because it started to block > Spark > >>> PRs time to time. Since the number > >>> of PRs grows hugely in Spark community, this is critical to not block > >>> other PRs. > >>> > >>> There has been a problem at CRAN (See > >>> https://github.com/apache/spark/pull/20005 for analysis). > >>> To cut it short, the root cause is malformed package info from > >>> https://cran.r-project.org/src/contrib/PACKAGES > >>> from server side, and this had to be fixed by requesting it to CRAN > >>> sysaadmin's help. > >>> > >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am > >>> pretty sure it's the same issue > >>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2 > >>> times > >>> https://issues.apache.org/jira/browse/SPARK-22812 > >>> > >>> This happened 5 times for roughly about 10 months, causing blocking > >>> almost all PRs in Apache Spark. > >>> Historically, it blocked whole PRs for few days once, and whole Spark > >>> community had to stop working. > >>> > >>> I assume this has been not a super big big issue so far for other > >>> projects or other people because apparently > >>> higher version of R has some logics to handle this malformed documents > >>> (at least I verified R 3.4.0 works fine). > >>> > >>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated > >>> from what I have seen before), > >>> which is unable to parse the malformed server's response. > >>> > >>> So, I want to talk about how we are going to handle this. Possible > >>> solutions are: > >>> > >>> 1. We should start a talk with CRAN sysadmin to permanently prevent > this > >>> issue > >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to > test > >>> low R versions) > >>> 3. ... > >>> > >>> If if we fine, I would like to suggest to forward this email to CRAN > >>> sysadmin to discuss further about this. > >>> > >>> Adding Liang-Chi Felix and Shivaram who I already talked about this few > >>> times before. > >>> > >>> Thanks all. > >>> > >>> > >>> > >>> > > > > > > -- > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
Re: [discuss] SparkR CRAN feasibility check server problem
It’s a great point about min R version. From what I see, mostly because of fixes and packages support, most users of R are fairly up to date? So perhaps 3.4 as min version is reasonable esp. for Spark 3. Are we getting traction with CRAN sysadmin? It seems like this has been broken a few times. From: Liang-Chi Hsieh Sent: Saturday, November 10, 2018 2:32 AM To: dev@spark.apache.org Subject: Re: [discuss] SparkR CRAN feasibility check server problem Yeah, thanks Hyukjin Kwon for bringing this up for discussion. I don't know how higher versions of R are widely used across R community. If R version 3.1.x was not very commonly used, I think we can discuss to upgrade minimum R version in next Spark version. If we ended up with not upgrading, we can discuss with CRAN sysadmin to fix it by the service side automatically that prevents malformed R packages info. So we don't need to fix it manually every time. Hyukjin Kwon wrote >> Can upgrading R able to fix the issue. Is this perhaps not necessarily > malform but some new format for new versions perhaps? > That's my guess. I am not totally sure about it tho. > >> Anyway we should consider upgrading R version if that fixes the problem. > Yea, we should. If we should, it should be more them R 3.4. Maybe it's > good > time to start to talk about minimum R version. 3.1.x is too old. It's > released 4.5 years ago. > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0, > deprecating lower versions, bumping up R to 3.4 might be reasonable > option. > > Adding Shane as well. > > If we ended up with not upgrading it, I will forward this email to CRAN > sysadmin to discuss further anyway. > > > > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung < > felixcheung@ > >님이 작성: > >> Thanks for being this up and much appreciate with keeping on top of this >> at all times. >> >> Can upgrading R able to fix the issue. Is this perhaps not necessarily >> malform but some new format for new versions perhaps? Anyway we should >> consider upgrading R version if that fixes the problem. >> >> As an option we could also disable the repo check in Jenkins but I can >> see >> that could also be problematic. >> >> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon < > gurwls223@ > > wrote: >> >>> Hi all, >>> >>> I want to raise the CRAN failure issue because it started to block Spark >>> PRs time to time. Since the number >>> of PRs grows hugely in Spark community, this is critical to not block >>> other PRs. >>> >>> There has been a problem at CRAN (See >>> https://github.com/apache/spark/pull/20005 for analysis). >>> To cut it short, the root cause is malformed package info from >>> https://cran.r-project.org/src/contrib/PACKAGES >>> from server side, and this had to be fixed by requesting it to CRAN >>> sysaadmin's help. >>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am >>> pretty sure it's the same issue >>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2 >>> times >>> https://issues.apache.org/jira/browse/SPARK-22812 >>> >>> This happened 5 times for roughly about 10 months, causing blocking >>> almost all PRs in Apache Spark. >>> Historically, it blocked whole PRs for few days once, and whole Spark >>> community had to stop working. >>> >>> I assume this has been not a super big big issue so far for other >>> projects or other people because apparently >>> higher version of R has some logics to handle this malformed documents >>> (at least I verified R 3.4.0 works fine). >>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated >>> from what I have seen before), >>> which is unable to parse the malformed server's response. >>> >>> So, I want to talk about how we are going to handle this. Possible >>> solutions are: >>> >>> 1. We should start a talk with CRAN sysadmin to permanently prevent this >>> issue >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test >>> low R versions) >>> 3. ... >>> >>> If if we fine, I would like to suggest to forward this email to CRAN >>> sysadmin to discuss further about this. >>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about this few >>> times before. >>> >>> Thanks all. >>> >>> >>> >>> -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [discuss] SparkR CRAN feasibility check server problem
Yeah, thanks Hyukjin Kwon for bringing this up for discussion. I don't know how higher versions of R are widely used across R community. If R version 3.1.x was not very commonly used, I think we can discuss to upgrade minimum R version in next Spark version. If we ended up with not upgrading, we can discuss with CRAN sysadmin to fix it by the service side automatically that prevents malformed R packages info. So we don't need to fix it manually every time. Hyukjin Kwon wrote >> Can upgrading R able to fix the issue. Is this perhaps not necessarily > malform but some new format for new versions perhaps? > That's my guess. I am not totally sure about it tho. > >> Anyway we should consider upgrading R version if that fixes the problem. > Yea, we should. If we should, it should be more them R 3.4. Maybe it's > good > time to start to talk about minimum R version. 3.1.x is too old. It's > released 4.5 years ago. > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0, > deprecating lower versions, bumping up R to 3.4 might be reasonable > option. > > Adding Shane as well. > > If we ended up with not upgrading it, I will forward this email to CRAN > sysadmin to discuss further anyway. > > > > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung < > felixcheung@ > >님이 작성: > >> Thanks for being this up and much appreciate with keeping on top of this >> at all times. >> >> Can upgrading R able to fix the issue. Is this perhaps not necessarily >> malform but some new format for new versions perhaps? Anyway we should >> consider upgrading R version if that fixes the problem. >> >> As an option we could also disable the repo check in Jenkins but I can >> see >> that could also be problematic. >> >> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon < > gurwls223@ > > wrote: >> >>> Hi all, >>> >>> I want to raise the CRAN failure issue because it started to block Spark >>> PRs time to time. Since the number >>> of PRs grows hugely in Spark community, this is critical to not block >>> other PRs. >>> >>> There has been a problem at CRAN (See >>> https://github.com/apache/spark/pull/20005 for analysis). >>> To cut it short, the root cause is malformed package info from >>> https://cran.r-project.org/src/contrib/PACKAGES >>> from server side, and this had to be fixed by requesting it to CRAN >>> sysaadmin's help. >>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am >>> pretty sure it's the same issue >>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2 >>> times >>> https://issues.apache.org/jira/browse/SPARK-22812 >>> >>> This happened 5 times for roughly about 10 months, causing blocking >>> almost all PRs in Apache Spark. >>> Historically, it blocked whole PRs for few days once, and whole Spark >>> community had to stop working. >>> >>> I assume this has been not a super big big issue so far for other >>> projects or other people because apparently >>> higher version of R has some logics to handle this malformed documents >>> (at least I verified R 3.4.0 works fine). >>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated >>> from what I have seen before), >>> which is unable to parse the malformed server's response. >>> >>> So, I want to talk about how we are going to handle this. Possible >>> solutions are: >>> >>> 1. We should start a talk with CRAN sysadmin to permanently prevent this >>> issue >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test >>> low R versions) >>> 3. ... >>> >>> If if we fine, I would like to suggest to forward this email to CRAN >>> sysadmin to discuss further about this. >>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about this few >>> times before. >>> >>> Thanks all. >>> >>> >>> >>> -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [discuss] SparkR CRAN feasibility check server problem
> Can upgrading R able to fix the issue. Is this perhaps not necessarily malform but some new format for new versions perhaps? That's my guess. I am not totally sure about it tho. > Anyway we should consider upgrading R version if that fixes the problem. Yea, we should. If we should, it should be more them R 3.4. Maybe it's good time to start to talk about minimum R version. 3.1.x is too old. It's released 4.5 years ago. R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0, deprecating lower versions, bumping up R to 3.4 might be reasonable option. Adding Shane as well. If we ended up with not upgrading it, I will forward this email to CRAN sysadmin to discuss further anyway. 2018년 11월 2일 (금) 오후 12:51, Felix Cheung 님이 작성: > Thanks for being this up and much appreciate with keeping on top of this > at all times. > > Can upgrading R able to fix the issue. Is this perhaps not necessarily > malform but some new format for new versions perhaps? Anyway we should > consider upgrading R version if that fixes the problem. > > As an option we could also disable the repo check in Jenkins but I can see > that could also be problematic. > > > On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon wrote: > >> Hi all, >> >> I want to raise the CRAN failure issue because it started to block Spark >> PRs time to time. Since the number >> of PRs grows hugely in Spark community, this is critical to not block >> other PRs. >> >> There has been a problem at CRAN (See >> https://github.com/apache/spark/pull/20005 for analysis). >> To cut it short, the root cause is malformed package info from >> https://cran.r-project.org/src/contrib/PACKAGES >> from server side, and this had to be fixed by requesting it to CRAN >> sysaadmin's help. >> >> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am >> pretty sure it's the same issue >> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2 >> times >> https://issues.apache.org/jira/browse/SPARK-22812 >> >> This happened 5 times for roughly about 10 months, causing blocking >> almost all PRs in Apache Spark. >> Historically, it blocked whole PRs for few days once, and whole Spark >> community had to stop working. >> >> I assume this has been not a super big big issue so far for other >> projects or other people because apparently >> higher version of R has some logics to handle this malformed documents >> (at least I verified R 3.4.0 works fine). >> >> For our side, Jenkins has low R version (R 3.1.1 if that's not updated >> from what I have seen before), >> which is unable to parse the malformed server's response. >> >> So, I want to talk about how we are going to handle this. Possible >> solutions are: >> >> 1. We should start a talk with CRAN sysadmin to permanently prevent this >> issue >> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test >> low R versions) >> 3. ... >> >> If if we fine, I would like to suggest to forward this email to CRAN >> sysadmin to discuss further about this. >> >> Adding Liang-Chi Felix and Shivaram who I already talked about this few >> times before. >> >> Thanks all. >> >> >> >>
Re: [discuss] SparkR CRAN feasibility check server problem
Thanks for being this up and much appreciate with keeping on top of this at all times. Can upgrading R able to fix the issue. Is this perhaps not necessarily malform but some new format for new versions perhaps? Anyway we should consider upgrading R version if that fixes the problem. As an option we could also disable the repo check in Jenkins but I can see that could also be problematic. On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon wrote: > Hi all, > > I want to raise the CRAN failure issue because it started to block Spark > PRs time to time. Since the number > of PRs grows hugely in Spark community, this is critical to not block > other PRs. > > There has been a problem at CRAN (See > https://github.com/apache/spark/pull/20005 for analysis). > To cut it short, the root cause is malformed package info from > https://cran.r-project.org/src/contrib/PACKAGES > from server side, and this had to be fixed by requesting it to CRAN > sysaadmin's help. > > https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am > pretty sure it's the same issue > https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2 > times > https://issues.apache.org/jira/browse/SPARK-22812 > > This happened 5 times for roughly about 10 months, causing blocking almost > all PRs in Apache Spark. > Historically, it blocked whole PRs for few days once, and whole Spark > community had to stop working. > > I assume this has been not a super big big issue so far for other projects > or other people because apparently > higher version of R has some logics to handle this malformed documents (at > least I verified R 3.4.0 works fine). > > For our side, Jenkins has low R version (R 3.1.1 if that's not updated > from what I have seen before), > which is unable to parse the malformed server's response. > > So, I want to talk about how we are going to handle this. Possible > solutions are: > > 1. We should start a talk with CRAN sysadmin to permanently prevent this > issue > 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test > low R versions) > 3. ... > > If if we fine, I would like to suggest to forward this email to CRAN > sysadmin to discuss further about this. > > Adding Liang-Chi Felix and Shivaram who I already talked about this few > times before. > > Thanks all. > > > >
[discuss] SparkR CRAN feasibility check server problem
Hi all, I want to raise the CRAN failure issue because it started to block Spark PRs time to time. Since the number of PRs grows hugely in Spark community, this is critical to not block other PRs. There has been a problem at CRAN (See https://github.com/apache/spark/pull/20005 for analysis). To cut it short, the root cause is malformed package info from https://cran.r-project.org/src/contrib/PACKAGES from server side, and this had to be fixed by requesting it to CRAN sysaadmin's help. https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am pretty sure it's the same issue https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2 times https://issues.apache.org/jira/browse/SPARK-22812 This happened 5 times for roughly about 10 months, causing blocking almost all PRs in Apache Spark. Historically, it blocked whole PRs for few days once, and whole Spark community had to stop working. I assume this has been not a super big big issue so far for other projects or other people because apparently higher version of R has some logics to handle this malformed documents (at least I verified R 3.4.0 works fine). For our side, Jenkins has low R version (R 3.1.1 if that's not updated from what I have seen before), which is unable to parse the malformed server's response. So, I want to talk about how we are going to handle this. Possible solutions are: 1. We should start a talk with CRAN sysadmin to permanently prevent this issue 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test low R versions) 3. ... If if we fine, I would like to suggest to forward this email to CRAN sysadmin to discuss further about this. Adding Liang-Chi Felix and Shivaram who I already talked about this few times before. Thanks all.