[jira] [Commented] (SPARK-15799) Release SparkR on CRAN
[
https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179829#comment-16179829
]
Brendan Dwyer commented on SPARK-15799:
---
I opened a [PR|https://github.com/apache/spark/pull/19342] that adds a
{{SystemRequirements}} field and adds an {{on.exit()}} call after {{attach}}
but if everyone is okay with using {{with}} I can close my PR.
> Release SparkR on CRAN
> --
>
> Key: SPARK-15799
> URL: https://issues.apache.org/jira/browse/SPARK-15799
> Project: Spark
> Issue Type: New Feature
> Components: SparkR
>Reporter: Xiangrui Meng
>
> Story: "As an R user, I would like to see SparkR released on CRAN, so I can
> use SparkR easily in an existing R environment and have other packages built
> on top of SparkR."
> I made this JIRA with the following questions in mind:
> * Are there known issues that prevent us releasing SparkR on CRAN?
> * Do we want to package Spark jars in the SparkR release?
> * Are there license issues?
> * How does it fit into Spark's release process?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15799) Release SparkR on CRAN
[
https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179747#comment-16179747
]
Brendan Dwyer commented on SPARK-15799:
---
The {{Description}} field has been fixed. {{SystemRequirements}} is an easy 1
line fix.
So I assume we need to discuses if we should change {{attach}} method to
{{SparkDataFrame}} or appeal to the CRAN people for an exception. Thoughts?
Here is the Good practice section referenced above:
{quote}
h2. Good practice
{{attach}} has the side effect of altering the search path and this can easily
lead to the wrong object of a particular name being found. People do often
forget to {{detach}} databases.
In interactive use, {{with}} is usually preferable to the use of attach/detach,
unless what is a {{save()}}-produced file in which case {{attach()}} is a
(safety) wrapper for {{load()}}.
In programming, functions should not change the search path unless that is
their purpose. Often {{with}} can be used within a function. If not, good
practice is to
* Always use a distinctive name argument, and
* To immediately follow the attach call by an on.exit call to detach using the
distinctive name.
This ensures that the search path is left unchanged even if the function is
interrupted or if code after the attach call changes the search path.
{quote}
> Release SparkR on CRAN
> --
>
> Key: SPARK-15799
> URL: https://issues.apache.org/jira/browse/SPARK-15799
> Project: Spark
> Issue Type: New Feature
> Components: SparkR
>Reporter: Xiangrui Meng
>
> Story: "As an R user, I would like to see SparkR released on CRAN, so I can
> use SparkR easily in an existing R environment and have other packages built
> on top of SparkR."
> I made this JIRA with the following questions in mind:
> * Are there known issues that prevent us releasing SparkR on CRAN?
> * Do we want to package Spark jars in the SparkR release?
> * Are there license issues?
> * How does it fit into Spark's release process?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15799) Release SparkR on CRAN
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114733#comment-16114733 ] Brendan Dwyer commented on SPARK-15799: --- [~felixcheung] awesome! Looking forward to this. > Release SparkR on CRAN > -- > > Key: SPARK-15799 > URL: https://issues.apache.org/jira/browse/SPARK-15799 > Project: Spark > Issue Type: New Feature > Components: SparkR >Reporter: Xiangrui Meng > > Story: "As an R user, I would like to see SparkR released on CRAN, so I can > use SparkR easily in an existing R environment and have other packages built > on top of SparkR." > I made this JIRA with the following questions in mind: > * Are there known issues that prevent us releasing SparkR on CRAN? > * Do we want to package Spark jars in the SparkR release? > * Are there license issues? > * How does it fit into Spark's release process? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15799) Release SparkR on CRAN
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113404#comment-16113404 ] Brendan Dwyer commented on SPARK-15799: --- Is there any update on this? > Release SparkR on CRAN > -- > > Key: SPARK-15799 > URL: https://issues.apache.org/jira/browse/SPARK-15799 > Project: Spark > Issue Type: New Feature > Components: SparkR >Reporter: Xiangrui Meng > > Story: "As an R user, I would like to see SparkR released on CRAN, so I can > use SparkR easily in an existing R environment and have other packages built > on top of SparkR." > I made this JIRA with the following questions in mind: > * Are there known issues that prevent us releasing SparkR on CRAN? > * Do we want to package Spark jars in the SparkR release? > * Are there license issues? > * How does it fit into Spark's release process? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio
[
https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100806#comment-16100806
]
Brendan Dwyer commented on SPARK-21532:
---
I've opened [an issue with
RStudio|https://support.rstudio.com/hc/en-us/community/posts/115007135727-SparkR-progress-bar-issues].
Should I close this JIRA or keep it open until this is fixed?
> Improve console progress bar in RStudio
> ---
>
> Key: SPARK-21532
> URL: https://issues.apache.org/jira/browse/SPARK-21532
> Project: Spark
> Issue Type: Improvement
> Components: SparkR
>Affects Versions: 2.2.0
> Environment: RStudio on Linux Mint, RStudio server on CentOS,
> probably RStudio on other OS's
>Reporter: Brendan Dwyer
>Priority: Trivial
>
> When a SparkR job is run through terminal the progress bar appears neatly and
> then disappears after the job is done however when a SparkR job is run in
> RStudio the progress bar gets printed to multiple lines and never goes away:
> {code}
> [Stage 38:> (0 + 3) /
> 8][Stage 38:===> (1 + 3)
> / 8][Stage 38:=>(3 +
> 3) / 8][Stage 38:=>(4
> + 3) / 8][Stage 38:===>
> (6 + 2) / 8][Stage 38:==>
> (7 + 1) / 8]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio
[
https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100695#comment-16100695
]
Brendan Dwyer commented on SPARK-21532:
---
https://support.rstudio.com/hc/en-us/community/posts/203136268-Carriage-return-does-not-work-on-stderr-
> Improve console progress bar in RStudio
> ---
>
> Key: SPARK-21532
> URL: https://issues.apache.org/jira/browse/SPARK-21532
> Project: Spark
> Issue Type: Improvement
> Components: SparkR
>Affects Versions: 2.2.0
> Environment: RStudio on Linux Mint, RStudio server on CentOS,
> probably RStudio on other OS's
>Reporter: Brendan Dwyer
>Priority: Trivial
>
> When a SparkR job is run through terminal the progress bar appears neatly and
> then disappears after the job is done however when a SparkR job is run in
> RStudio the progress bar gets printed to multiple lines and never goes away:
> {code}
> [Stage 38:> (0 + 3) /
> 8][Stage 38:===> (1 + 3)
> / 8][Stage 38:=>(3 +
> 3) / 8][Stage 38:=>(4
> + 3) / 8][Stage 38:===>
> (6 + 2) / 8][Stage 38:==>
> (7 + 1) / 8]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21532) Improve console progress bar in RStudio
[
https://issues.apache.org/jira/browse/SPARK-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100693#comment-16100693
]
Brendan Dwyer commented on SPARK-21532:
---
[~srowen] I assume so.
I think in it's current state the progress bar is more annoying than it is
useful in RStudio. Maybe we should set _spark.ui.showConsoleProgress_ to FALSE
by default if the session is started from RStudio?
> Improve console progress bar in RStudio
> ---
>
> Key: SPARK-21532
> URL: https://issues.apache.org/jira/browse/SPARK-21532
> Project: Spark
> Issue Type: Improvement
> Components: SparkR
>Affects Versions: 2.2.0
> Environment: RStudio on Linux Mint, RStudio server on CentOS,
> probably RStudio on other OS's
>Reporter: Brendan Dwyer
>Priority: Trivial
>
> When a SparkR job is run through terminal the progress bar appears neatly and
> then disappears after the job is done however when a SparkR job is run in
> RStudio the progress bar gets printed to multiple lines and never goes away:
> {code}
> [Stage 38:> (0 + 3) /
> 8][Stage 38:===> (1 + 3)
> / 8][Stage 38:=>(3 +
> 3) / 8][Stage 38:=>(4
> + 3) / 8][Stage 38:===>
> (6 + 2) / 8][Stage 38:==>
> (7 + 1) / 8]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-21532) Improve console progress bar in RStudio
Brendan Dwyer created SPARK-21532:
-
Summary: Improve console progress bar in RStudio
Key: SPARK-21532
URL: https://issues.apache.org/jira/browse/SPARK-21532
Project: Spark
Issue Type: Improvement
Components: SparkR
Affects Versions: 2.2.0
Environment: RStudio on Linux Mint, RStudio server on CentOS, probably
RStudio on other OS's
Reporter: Brendan Dwyer
Priority: Trivial
When a SparkR job is run through terminal the progress bar appears neatly and
then disappears after the job is done however when a SparkR job is run in
RStudio the progress bar gets printed to multiple lines and never goes away:
{code}
[Stage 38:> (0 + 3) /
8][Stage 38:===> (1 + 3) /
8][Stage 38:=>(3 + 3) /
8][Stage 38:=>(4 + 3) /
8][Stage 38:===> (6 + 2) /
8][Stage 38:==> (7 + 1) /
8]
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20298) Spelling mistake: charactor
[
https://issues.apache.org/jira/browse/SPARK-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964741#comment-15964741
]
Brendan Dwyer commented on SPARK-20298:
---
I can work on this
> Spelling mistake: charactor
> ---
>
> Key: SPARK-20298
> URL: https://issues.apache.org/jira/browse/SPARK-20298
> Project: Spark
> Issue Type: Bug
> Components: SparkR
>Affects Versions: 2.1.0
>Reporter: Brendan Dwyer
>Priority: Trivial
>
> "charactor" should be "character"
> {code}
> R/pkg/R/DataFrame.R:2821: stop("path should be charactor, NULL
> or omitted.")
> R/pkg/R/DataFrame.R:2828: stop("mode should be charactor or
> omitted. It is 'error' by default.")
> R/pkg/R/DataFrame.R:3043: stop("value should be an integer,
> numeric, charactor or named list.")
> R/pkg/R/DataFrame.R:3055: stop("Each item in value should be
> an integer, numeric or charactor.")
> R/pkg/R/DataFrame.R:3601: stop("outputMode should be charactor
> or omitted.")
> R/pkg/R/SQLContext.R:609:stop("path should be charactor, NULL or
> omitted.")
> R/pkg/inst/tests/testthat/test_sparkSQL.R:2929: "path should be
> charactor, NULL or omitted.")
> R/pkg/inst/tests/testthat/test_sparkSQL.R:2931: "mode should be
> charactor or omitted. It is 'error' by default.")
> R/pkg/inst/tests/testthat/test_sparkSQL.R:2950: "path should be
> charactor, NULL or omitted.")
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20298) Spelling mistake: charactor
Brendan Dwyer created SPARK-20298:
-
Summary: Spelling mistake: charactor
Key: SPARK-20298
URL: https://issues.apache.org/jira/browse/SPARK-20298
Project: Spark
Issue Type: Bug
Components: SparkR
Affects Versions: 2.1.0
Reporter: Brendan Dwyer
Priority: Trivial
"charactor" should be "character"
{code}
R/pkg/R/DataFrame.R:2821: stop("path should be charactor, NULL or
omitted.")
R/pkg/R/DataFrame.R:2828: stop("mode should be charactor or
omitted. It is 'error' by default.")
R/pkg/R/DataFrame.R:3043: stop("value should be an integer,
numeric, charactor or named list.")
R/pkg/R/DataFrame.R:3055: stop("Each item in value should be
an integer, numeric or charactor.")
R/pkg/R/DataFrame.R:3601: stop("outputMode should be charactor or
omitted.")
R/pkg/R/SQLContext.R:609:stop("path should be charactor, NULL or omitted.")
R/pkg/inst/tests/testthat/test_sparkSQL.R:2929: "path should be
charactor, NULL or omitted.")
R/pkg/inst/tests/testthat/test_sparkSQL.R:2931: "mode should be
charactor or omitted. It is 'error' by default.")
R/pkg/inst/tests/testthat/test_sparkSQL.R:2950: "path should be
charactor, NULL or omitted.")
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19387) CRAN tests do not run with SparkR source package
[ https://issues.apache.org/jira/browse/SPARK-19387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843493#comment-15843493 ] Brendan Dwyer commented on SPARK-19387: --- Should this be under [SPARK-15799]? > CRAN tests do not run with SparkR source package > > > Key: SPARK-19387 > URL: https://issues.apache.org/jira/browse/SPARK-19387 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 2.1.0 >Reporter: Felix Cheung >Assignee: Felix Cheung > > It looks like sparkR.session() is not installing Spark - as a result, running > R CMD check --as-cran SparkR_*.tar.gz fails, blocking possible submission to > CRAN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753093#comment-15753093
]
Brendan Dwyer edited comment on SPARK-18817 at 12/16/16 1:45 AM:
-
I'm not sure the CRAN people would be okay with that. It might be enough to
pass any automatic testing they have but it would still be against their
policies.
{quote}
Limited exceptions may be allowed in interactive sessions if the package
*obtains confirmation from the user*.
{quote}
was (Author: bdwyer):
I'm not sure the CRAN people would be okay with that. It might be enough to
pass any automatic testing they have but it would still be against their
policies.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753093#comment-15753093
]
Brendan Dwyer edited comment on SPARK-18817 at 12/16/16 1:30 AM:
-
I'm not sure the CRAN people would be okay with that. It might be enough to
pass any automatic testing they have but it would still be against their
policies.
was (Author: bdwyer):
I'm not sure the CRAN people would be okay with that.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753095#comment-15753095
]
Brendan Dwyer commented on SPARK-18817:
---
{code}
library("SparkR")
sparkR.session()
df <- as.DataFrame(iris)
{code}
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753093#comment-15753093
]
Brendan Dwyer commented on SPARK-18817:
---
I'm not sure the CRAN people would be okay with that.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752951#comment-15752951
]
Brendan Dwyer commented on SPARK-18817:
---
[~shivaram] it does not happen if I disable Hive.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752844#comment-15752844
]
Brendan Dwyer commented on SPARK-18817:
---
I'm also seeing _derby.log_ and a folder named _metastore_db_ being created in
my working directory when I run the following:
{code}
library("SparkR")
sparkR.session()
df <- as.DataFrame(iris)
{code}
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>Priority: Critical
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
[
https://issues.apache.org/jira/browse/SPARK-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brendan Dwyer updated SPARK-18817:
--
Description:
Per CRAN policies
https://cran.r-project.org/web/packages/policies.html
{quote}
- Packages should not write in the users’ home filespace, nor anywhere else on
the file system apart from the R session’s temporary directory (or during
installation in the location pointed to by TMPDIR: and such usage should be
cleaned up). Installing into the system’s R installation (e.g., scripts to its
bin directory) is not allowed.
Limited exceptions may be allowed in interactive sessions if the package
obtains confirmation from the user.
- Packages should not modify the global environment (user’s workspace).
{quote}
Currently "spark-warehouse" gets created in the working directory when
sparkR.session() is called.
was:
Per CRAN policies
https://cran.r-project.org/web/packages/policies.html
"Packages should not write in the users’ home filespace, nor anywhere else on
the file system apart from the R session’s temporary directory (or during
installation in the location pointed to by TMPDIR: and such usage should be
cleaned up). Installing into the system’s R installation (e.g., scripts to its
bin directory) is not allowed.
Limited exceptions may be allowed in interactive sessions if the package
obtains confirmation from the user.
- Packages should not modify the global environment (user’s workspace)."
Currently "spark-warehouse" gets created in the working directory when
sparkR.session() is called.
> Ensure nothing is written outside R's tempdir() by default
> --
>
> Key: SPARK-18817
> URL: https://issues.apache.org/jira/browse/SPARK-18817
> Project: Spark
> Issue Type: Sub-task
> Components: SparkR
>Reporter: Brendan Dwyer
>
> Per CRAN policies
> https://cran.r-project.org/web/packages/policies.html
> {quote}
> - Packages should not write in the users’ home filespace, nor anywhere else
> on the file system apart from the R session’s temporary directory (or during
> installation in the location pointed to by TMPDIR: and such usage should be
> cleaned up). Installing into the system’s R installation (e.g., scripts to
> its bin directory) is not allowed.
> Limited exceptions may be allowed in interactive sessions if the package
> obtains confirmation from the user.
> - Packages should not modify the global environment (user’s workspace).
> {quote}
> Currently "spark-warehouse" gets created in the working directory when
> sparkR.session() is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18817) Ensure nothing is written outside R's tempdir() by default
Brendan Dwyer created SPARK-18817: - Summary: Ensure nothing is written outside R's tempdir() by default Key: SPARK-18817 URL: https://issues.apache.org/jira/browse/SPARK-18817 Project: Spark Issue Type: Sub-task Components: SparkR Reporter: Brendan Dwyer Per CRAN policies https://cran.r-project.org/web/packages/policies.html "Packages should not write in the users’ home filespace, nor anywhere else on the file system apart from the R session’s temporary directory (or during installation in the location pointed to by TMPDIR: and such usage should be cleaned up). Installing into the system’s R installation (e.g., scripts to its bin directory) is not allowed. Limited exceptions may be allowed in interactive sessions if the package obtains confirmation from the user. - Packages should not modify the global environment (user’s workspace)." Currently "spark-warehouse" gets created in the working directory when sparkR.session() is called. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15799) Release SparkR on CRAN
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736657#comment-15736657 ] Brendan Dwyer commented on SPARK-15799: --- I think that's the best solution. Can I create a pull request to make this change? > Release SparkR on CRAN > -- > > Key: SPARK-15799 > URL: https://issues.apache.org/jira/browse/SPARK-15799 > Project: Spark > Issue Type: New Feature > Components: SparkR >Reporter: Xiangrui Meng > > Story: "As an R user, I would like to see SparkR released on CRAN, so I can > use SparkR easily in an existing R environment and have other packages built > on top of SparkR." > I made this JIRA with the following questions in mind: > * Are there known issues that prevent us releasing SparkR on CRAN? > * Do we want to package Spark jars in the SparkR release? > * Are there license issues? > * How does it fit into Spark's release process? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15799) Release SparkR on CRAN
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736591#comment-15736591 ] Brendan Dwyer commented on SPARK-15799: --- On my machine it gets created in my home directory when I call sparkR.session() > Release SparkR on CRAN > -- > > Key: SPARK-15799 > URL: https://issues.apache.org/jira/browse/SPARK-15799 > Project: Spark > Issue Type: New Feature > Components: SparkR >Reporter: Xiangrui Meng > > Story: "As an R user, I would like to see SparkR released on CRAN, so I can > use SparkR easily in an existing R environment and have other packages built > on top of SparkR." > I made this JIRA with the following questions in mind: > * Are there known issues that prevent us releasing SparkR on CRAN? > * Do we want to package Spark jars in the SparkR release? > * Are there license issues? > * How does it fit into Spark's release process? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15799) Release SparkR on CRAN
[
https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736572#comment-15736572
]
Brendan Dwyer commented on SPARK-15799:
---
[CRAN policy|https://cran.r-project.org/web/packages/policies.html] states:
{quote}
- Packages should not write in the users’ home filespace, nor anywhere else on
the file system apart from the R session’s temporary directory (or during
installation in the location pointed to by TMPDIR: and such usage should be
cleaned up). Installing into the system’s R installation (e.g., scripts to its
bin directory) is not allowed.
Limited exceptions may be allowed in interactive sessions if the package
obtains confirmation from the user.
- Packages should not modify the global environment (user’s workspace).
{quote}
Do we need to move the location of spark-warehouse to a temporary directory?
> Release SparkR on CRAN
> --
>
> Key: SPARK-15799
> URL: https://issues.apache.org/jira/browse/SPARK-15799
> Project: Spark
> Issue Type: New Feature
> Components: SparkR
>Reporter: Xiangrui Meng
>
> Story: "As an R user, I would like to see SparkR released on CRAN, so I can
> use SparkR easily in an existing R environment and have other packages built
> on top of SparkR."
> I made this JIRA with the following questions in mind:
> * Are there known issues that prevent us releasing SparkR on CRAN?
> * Do we want to package Spark jars in the SparkR release?
> * Are there license issues?
> * How does it fit into Spark's release process?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
