[spark] 01/01: change SparkR maintainer
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch r-maintainer in repository https://gitbox.apache.org/repos/asf/spark.git commit 5dbe3bbc5d29e2ba6caca022c37eb0b5a2f13202 Author: Felix Cheung AuthorDate: Sun May 23 16:49:27 2021 -0700 change SparkR maintainer --- R/pkg/DESCRIPTION | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION index f55286d..4a310e6 100644 --- a/R/pkg/DESCRIPTION +++ b/R/pkg/DESCRIPTION @@ -3,11 +3,11 @@ Type: Package Version: 3.2.0 Title: R Front End for 'Apache Spark' Description: Provides an R Front end for 'Apache Spark' <https://spark.apache.org>. -Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"), +Authors@R: c(person("Shivaram", "Venkataraman", role = "aut", email = "shiva...@cs.berkeley.edu"), person("Xiangrui", "Meng", role = "aut", email = "m...@databricks.com"), - person("Felix", "Cheung", role = "aut", + person("Felix", "Cheung", role = c("aut", "cre"), email = "felixche...@apache.org"), person(family = "The Apache Software Foundation", role = c("aut", "cph"))) License: Apache License (== 2.0) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch r-maintainer created (now 5dbe3bb)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a change to branch r-maintainer in repository https://gitbox.apache.org/repos/asf/spark.git. at 5dbe3bb change SparkR maintainer This branch includes the following new commits: new 5dbe3bb change SparkR maintainer The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Add test-hadoop3.2 phrase (#203)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 16052cb Add test-hadoop3.2 phrase (#203) 16052cb is described below commit 16052cbd63d17ce4d74ee2ca302e0a8532d46e80 Author: Yuming Wang AuthorDate: Mon May 13 06:53:29 2019 +0800 Add test-hadoop3.2 phrase (#203) --- developer-tools.md | 1 + 1 file changed, 1 insertion(+) diff --git a/developer-tools.md b/developer-tools.md index e5a66d8..21abf47 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -254,6 +254,7 @@ your pull request to change testing behavior. This includes: - `[test-maven]` - signals to test the pull request using maven - `[test-hadoop2.7]` - signals to test using Spark's Hadoop 2.7 profile +- `[test-hadoop3.2]` - signals to test using Spark's Hadoop 3.2 profile Binary compatibility - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[zeppelin] branch master updated: [ZEPPELIN-4026] document anonymous access
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new 05fdfe5 [ZEPPELIN-4026] document anonymous access 05fdfe5 is described below commit 05fdfe5162bb9f38eb927fc68d754370b316a133 Author: Felix Cheung AuthorDate: Sun Mar 3 23:42:11 2019 -0500 [ZEPPELIN-4026] document anonymous access ### What is this PR for? doc update ### What type of PR is it? [Documentation] ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-4026 ### Questions: * Does the licenses files need update? N * Is there breaking changes for older versions? N * Does this needs documentation? N --- docs/setup/security/shiro_authentication.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/setup/security/shiro_authentication.md b/docs/setup/security/shiro_authentication.md index c9bf905..40b6e53 100644 --- a/docs/setup/security/shiro_authentication.md +++ b/docs/setup/security/shiro_authentication.md @@ -28,6 +28,9 @@ limitations under the License. When you connect to Apache Zeppelin, you will be asked to enter your credentials. Once you logged in, then you have access to all notes including other user's notes. +## Important Note +By default, Zeppelin allows anonymous access. It is strongly recommended that you consider setting up Apache Shiro for authentication (as described in this document, see 2 Secure the Websocket channel), or only deploy and use Zeppelin in a secured and trusted environment. + ## Security Setup You can setup **Zeppelin notebook authentication** in some simple steps.
[zeppelin] branch master updated: [ZEPPELIN-4012] Save paragraphs after clone
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new dd624a6 [ZEPPELIN-4012] Save paragraphs after clone dd624a6 is described below commit dd624a67a4cf27e1778be260d964abc3df96e345 Author: Egor Klimov AuthorDate: Wed Feb 27 08:25:23 2019 +0300 [ZEPPELIN-4012] Save paragraphs after clone ### What is this PR for? Paragraphs in cloned note disappear after zeppelin restart if they weren't executed * Bug: ![clone_bug](https://user-images.githubusercontent.com/6136993/52918382-5ed02e00-3307-11e9-888e-5eb4dce756f1.gif) * Fix: ![clone_fix](https://user-images.githubusercontent.com/6136993/52918383-6263b500-3307-11e9-83e3-e851f65c34b8.gif) ### What type of PR is it? Bug Fix ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4012 ### How should this be tested? * CI pass ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No --- .../src/main/java/org/apache/zeppelin/notebook/Notebook.java | 1 + 1 file changed, 1 insertion(+) diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java index ccb9b95..ab32059 100644 --- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java +++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java @@ -244,6 +244,7 @@ public class Notebook { for (Paragraph p : paragraphs) { newNote.addCloneParagraph(p, subject); } +saveNote(newNote, subject); return newNote; }
[zeppelin] branch master updated: [ZEPPELIN-4011] Fix note path
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new 6dcc2ab [ZEPPELIN-4011] Fix note path 6dcc2ab is described below commit 6dcc2ab49c1f3d0c9ad4c8d68b005e37690faa7a Author: Egor Klimov AuthorDate: Wed Feb 27 08:24:01 2019 +0300 [ZEPPELIN-4011] Fix note path ### What is this PR for? Now note may move to the root directory after enabling cron in it. It caused by `Note#setName` * Bug: ![notepath](https://user-images.githubusercontent.com/6136993/52917693-87542a00-32ff-11e9-9fb1-82d21c0942f8.gif) * Fix: ![notepath_fix](https://user-images.githubusercontent.com/6136993/52917704-abb00680-32ff-11e9-8f50-0e8908e4edc0.gif) ### What type of PR is it? Bug Fix ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4011 ### How should this be tested? * CI pass ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No --- zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java index 403db06..2632e7d 100644 --- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java +++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java @@ -232,7 +232,7 @@ public class Note implements JsonSerializable { this.path = "/" + name; } } else { - int pos = this.path.indexOf("/"); + int pos = this.path.lastIndexOf("/"); this.path = this.path.substring(0, pos + 1) + this.name; } }
[spark] branch branch-2.3 updated: [R][BACKPORT-2.4] update package description
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new 6691c04 [R][BACKPORT-2.4] update package description 6691c04 is described below commit 6691c041dcab4c19d362aaff74f56e5beeda85cd Author: Felix Cheung AuthorDate: Thu Feb 21 08:42:15 2019 -0800 [R][BACKPORT-2.4] update package description doesn't port cleanly to 2.4. we need this in branch-2.4 and branch-2.3 Author: Felix Cheung Closes #23860 from felixcheung/2.4rdesc. (cherry picked from commit d8576301fd1d33675a9542791e58e7963081ce04) Signed-off-by: Felix Cheung --- R/pkg/DESCRIPTION | 6 ++ 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION index a82446e..136d782 100644 --- a/R/pkg/DESCRIPTION +++ b/R/pkg/DESCRIPTION @@ -1,8 +1,6 @@ Package: SparkR Type: Package Version: 2.3.4 -Title: R Frontend for Apache Spark -Description: Provides an R Frontend for Apache Spark. Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"), email = "shiva...@cs.berkeley.edu"), person("Xiangrui", "Meng", role = "aut", @@ -11,8 +9,8 @@ Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"), email = "felixche...@apache.org"), person(family = "The Apache Software Foundation", role = c("aut", "cph"))) License: Apache License (== 2.0) -URL: http://www.apache.org/ http://spark.apache.org/ -BugReports: http://spark.apache.org/contributing.html +URL: https://www.apache.org/ https://spark.apache.org/ +BugReports: https://spark.apache.org/contributing.html SystemRequirements: Java (== 8) Depends: R (>= 3.0), - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [R][BACKPORT-2.4] update package description
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new d8576301 [R][BACKPORT-2.4] update package description d8576301 is described below commit d8576301fd1d33675a9542791e58e7963081ce04 Author: Felix Cheung AuthorDate: Thu Feb 21 08:42:15 2019 -0800 [R][BACKPORT-2.4] update package description #23852 doesn't port cleanly to 2.4. we need this in branch-2.4 and branch-2.3 Author: Felix Cheung Closes #23860 from felixcheung/2.4rdesc. --- R/pkg/DESCRIPTION | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION index 2361289..5e3d186 100644 --- a/R/pkg/DESCRIPTION +++ b/R/pkg/DESCRIPTION @@ -1,8 +1,8 @@ Package: SparkR Type: Package Version: 2.4.2 -Title: R Frontend for Apache Spark -Description: Provides an R Frontend for Apache Spark. +Title: R Front end for 'Apache Spark' +Description: Provides an R Front end for 'Apache Spark' <https://spark.apache.org>. Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"), email = "shiva...@cs.berkeley.edu"), person("Xiangrui", "Meng", role = "aut", @@ -11,8 +11,8 @@ Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"), email = "felixche...@apache.org"), person(family = "The Apache Software Foundation", role = c("aut", "cph"))) License: Apache License (== 2.0) -URL: http://www.apache.org/ http://spark.apache.org/ -BugReports: http://spark.apache.org/contributing.html +URL: https://www.apache.org/ https://spark.apache.org/ +BugReports: https://spark.apache.org/contributing.html SystemRequirements: Java (== 8) Depends: R (>= 3.0), - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Update Thomas Graves Information (#179)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 6205847 Update Thomas Graves Information (#179) 6205847 is described below commit 6205847f851303dd8eee80dd25236a875ca3256a Author: Thomas Graves AuthorDate: Tue Feb 12 01:29:42 2019 -0600 Update Thomas Graves Information (#179) * Update Thomas Graves Information Update my company info. I ran jekyll build and server and it looked fine. * capitalize --- committers.md| 2 +- site/committers.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/committers.md b/committers.md index c5c4104..97b21a4 100644 --- a/committers.md +++ b/committers.md @@ -26,7 +26,7 @@ navigation: |Robert Evans|Oath| |Wenchen Fan|Databricks| |Joseph Gonzalez|UC Berkeley| -|Thomas Graves|Oath| +|Thomas Graves|NVIDIA| |Stephen Haberman|LinkedIn| |Mark Hamstra|ClearStory Data| |Seth Hendrickson|Cloudera| diff --git a/site/committers.html b/site/committers.html index 42c9171..a690ca7 100644 --- a/site/committers.html +++ b/site/committers.html @@ -276,7 +276,7 @@ Thomas Graves - Oath + NVIDIA Stephen Haberman - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[zeppelin] branch branch-0.7 updated: formatting fix for LDAP realm documentation
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch branch-0.7 in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/branch-0.7 by this push: new 7906a5c formatting fix for LDAP realm documentation 7906a5c is described below commit 7906a5c9443a33f4a21e3e93dccf06844649ce7f Author: Jake Maul <1141442+superawes...@users.noreply.github.com> AuthorDate: Sat Feb 9 22:37:26 2019 -0700 formatting fix for LDAP realm documentation spacing of "```" block causing it to be ignored, and the rest of the contents to be rendered incorrectly. ### What is this PR for? The LDAP section of the documentation for zeppelin-0.7.3 renders incorrectly, here: http://zeppelin.apache.org/docs/0.7.3/security/shiroauthentication.html It looks correct on GitHub, but not on the standalone site. This should fix it. ### What type of PR is it? [Documentation | Bug Fix] ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? none known * Does this needs documentation? no --- docs/security/shiroauthentication.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/security/shiroauthentication.md b/docs/security/shiroauthentication.md index 75a52e1..bf5e8e3 100644 --- a/docs/security/shiroauthentication.md +++ b/docs/security/shiroauthentication.md @@ -143,7 +143,8 @@ ldapRealm.contextFactory.authenticationMechanism = simple The other more flexible option is to use the LdapRealm. It allows for mapping of ldapgroups to roles and also allows for role/group based authentication into the zeppelin server. Sample configuration for this realm is given below. - ``` + +``` [main] ldapRealm=org.apache.zeppelin.realm.LdapRealm @@ -179,7 +180,7 @@ ldapRealm.allowedRolesForAuthentication = admin_role,user_role ldapRealm.permissionsByRole= user_role = *:ToDoItemsJdo:*:*, *:ToDoItem:*:*; admin_role = * securityManager.sessionManager = $sessionManager securityManager.realms = $ldapRealm - ``` +``` ### PAM [PAM](https://en.wikipedia.org/wiki/Pluggable_authentication_module) authentication support allows the reuse of existing authentication
[spark] branch master updated: [SPARK-26603][K8S] Update minikube backend
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 196ca0c [SPARK-26603][K8S] Update minikube backend 196ca0c is described below commit 196ca0c8f5f5caa2ae588ff4a63ce314aa42aecc Author: Stavros Kontopoulos AuthorDate: Sun Feb 3 17:15:20 2019 -0800 [SPARK-26603][K8S] Update minikube backend ## What changes were proposed in this pull request? - Covers latest minikube versions. - keeps the older version support Note: While I was facing disk pressure issues locally on machine, I noticed minikube status command would report that everything was working fine even if some kube-system pods were not up. I don't think the output is 100% reliable but it is good enough for most cases. ## How was this patch tested? Run it against latest version of minikube (v0.32.0). Author: Stavros Kontopoulos Closes #23520 from skonto/update-mini-backend. --- .../backend/minikube/Minikube.scala| 64 ++ 1 file changed, 54 insertions(+), 10 deletions(-) diff --git a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala index 6494cbc..58aa177 100644 --- a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala +++ b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala @@ -16,7 +16,6 @@ */ package org.apache.spark.deploy.k8s.integrationtest.backend.minikube -import java.io.File import java.nio.file.Paths import io.fabric8.kubernetes.client.{ConfigBuilder, DefaultKubernetesClient} @@ -26,8 +25,14 @@ import org.apache.spark.internal.Logging // TODO support windows private[spark] object Minikube extends Logging { - private val MINIKUBE_STARTUP_TIMEOUT_SECONDS = 60 + private val HOST_PREFIX = "host:" + private val KUBELET_PREFIX = "kubelet:" + private val APISERVER_PREFIX = "apiserver:" + private val KUBECTL_PREFIX = "kubectl:" + private val MINIKUBE_VM_PREFIX = "minikubeVM: " + private val MINIKUBE_PREFIX = "minikube: " + private val MINIKUBE_PATH = ".minikube" def getMinikubeIp: String = { val outputs = executeMinikube("ip") @@ -38,12 +43,21 @@ private[spark] object Minikube extends Logging { def getMinikubeStatus: MinikubeStatus.Value = { val statusString = executeMinikube("status") - .filter(line => line.contains("minikubeVM: ") || line.contains("minikube:")) - .head - .replaceFirst("minikubeVM: ", "") - .replaceFirst("minikube: ", "") -MinikubeStatus.unapply(statusString) +logInfo(s"Minikube status command output:\n$statusString") +// up to minikube version v0.30.0 use this to check for minikube status +val oldMinikube = statusString + .filter(line => line.contains(MINIKUBE_VM_PREFIX) || line.contains(MINIKUBE_PREFIX)) + +if (oldMinikube.isEmpty) { + getIfNewMinikubeStatus(statusString) +} else { + val finalStatusString = oldMinikube +.head +.replaceFirst(MINIKUBE_VM_PREFIX, "") +.replaceFirst(MINIKUBE_PREFIX, "") + MinikubeStatus.unapply(finalStatusString) .getOrElse(throw new IllegalStateException(s"Unknown status $statusString")) +} } def getKubernetesClient: DefaultKubernetesClient = { @@ -52,13 +66,43 @@ private[spark] object Minikube extends Logging { val kubernetesConf = new ConfigBuilder() .withApiVersion("v1") .withMasterUrl(kubernetesMaster) - .withCaCertFile(Paths.get(userHome, ".minikube", "ca.crt").toFile.getAbsolutePath) - .withClientCertFile(Paths.get(userHome, ".minikube", "apiserver.crt").toFile.getAbsolutePath) - .withClientKeyFile(Paths.get(userHome, ".minikube", "apiserver.key").toFile.getAbsolutePath) + .withCaCertFile(Paths.get(userHome, MINIKUBE_PATH, "ca.crt").toFile.getAbsolutePath) + .withClientCertFile(Paths.get(userHome, MINIKUBE_PATH, "apiserver.crt").toFile.getAbsolutePath) + .withClientKeyFile(Paths.get(userHome, MINIKUBE_PATH, "apiserver.key").toFile.getAbsolutePath) .build() new DefaultKubernetesClient(kubernetesConf) } + // Covers minikube status output after Minikube V
[zeppelin] branch master updated: [ZEPPELIN-3957] Divide SparkIntegrationTest into parts for build under 50 minutes
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new f7df680 [ZEPPELIN-3957] Divide SparkIntegrationTest into parts for build under 50 minutes f7df680 is described below commit f7df680ef91a2862e22fe914b8c6ddeae2491fc5 Author: Andrey Koshkin <37596468+d...@users.noreply.github.com> AuthorDate: Thu Jan 24 06:51:16 2019 +0300 [ZEPPELIN-3957] Divide SparkIntegrationTest into parts for build under 50 minutes ### What is this PR for? Fix Travis profiles and tests to archive build time under 50 minutes. ### What type of PR is it? [Improvement] ### What is the Jira issue? * [ZEPPELIN-3957] ### How should this be tested? * Travis CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no --- .travis.yml| 10 +++-- .../zeppelin/interpreter/SparkIntegrationTest.java | 16 ++ .../interpreter/SparkIntegrationTestPt1.java | 25 ++ .../interpreter/SparkIntegrationTestPt2.java | 25 ++ 4 files changed, 60 insertions(+), 16 deletions(-) diff --git a/.travis.yml b/.travis.yml index 2d810bf..ada776e 100644 --- a/.travis.yml +++ b/.travis.yml @@ -92,11 +92,17 @@ matrix: dist: trusty env: BUILD_PLUGINS="true" PYTHON="2" PROFILE="-Pspark-2.2" SPARKR="true" BUILD_FLAG="install -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl zeppelin-zengine,zeppelin-server,spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=ZeppelinSparkClusterTest,org.apache.zeppelin.spark.* -DfailIfNoTests=false" -# Integration test of spark interpreter with different spark versions under python3, only run SparkIntegrationTest. Also run spark unit test of spark 1.6 in this build. +# Integration test of spark interpreter with different spark versions under python3, only run SparkIntegrationTestPt1. Also run spark unit test of spark 1.6 in this build. - sudo: required jdk: "oraclejdk8" dist: trusty - env: BUILD_PLUGINS="true" PYTHON="3" SCALA_VER="2.10" PROFILE="-Pspark-1.6 -Pscala-2.10" SPARKR="true" BUILD_FLAG="install -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl zeppelin-zengine,spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=SparkIntegrationTest,org.apache.zeppelin.spark.* -DfailIfNoTests=false" + env: BUILD_PLUGINS="true" PYTHON="3" SCALA_VER="2.10" PROFILE="-Pspark-1.6 -Pscala-2.10" SPARKR="true" BUILD_FLAG="install -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl zeppelin-zengine,spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=SparkIntegrationTestPt1,org.apache.zeppelin.spark.* -DfailIfNoTests=false" + +# Integration test of spark interpreter with different spark versions under python3, only run SparkIntegrationTestPt2. Also run spark unit test of spark 1.6 in this build. +- sudo: required + jdk: "oraclejdk8" + dist: trusty + env: BUILD_PLUGINS="true" PYTHON="3" SCALA_VER="2.10" PROFILE="-Pspark-1.6 -Pscala-2.10" SPARKR="true" BUILD_FLAG="install -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl zeppelin-zengine,spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=SparkIntegrationTestPt2,org.apache.zeppelin.spark.* -DfailIfNoTests=false" # Test spark module for 2.4.0 with scala 2.11 - jdk: "oraclejdk8" diff --git a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/SparkIntegrationTest.java b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/SparkIntegrationTest.java index 8d076e3..0665953 100644 --- a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/SparkIntegrationTest.java +++ b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/SparkIntegrationTest.java @@ -21,8 +21,8 @@ import java.util.List; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; -@RunWith(value = Parameterized.class) -public class SparkIntegrationTest { + +public abstract class SparkIntegrationTest { private static Logger LOGGER = LoggerFactory.getLogger(SparkIntegrationTest.class); private stat
[zeppelin] branch master updated: [ZEPPELIN-3882] Neo4jInterpreter - Support Point and Date Types (#3284)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new 9a185f1 [ZEPPELIN-3882] Neo4jInterpreter - Support Point and Date Types (#3284) 9a185f1 is described below commit 9a185f19124259f0d510a3669d1023065bdf6197 Author: Andrea Santurbano AuthorDate: Mon Jan 21 20:51:01 2019 +0100 [ZEPPELIN-3882] Neo4jInterpreter - Support Point and Date Types (#3284) ### What is this PR for? Add the support for Point and Date data types introduced since Neo4j 3.4 ### What type of PR is it? [Improvement] In order to allow users to use the Neo4j Interpreter with the last introduced Data types ### Todos * [x] - Bumped Neo4j dependecies version * [x] - Added new data types to test ### What is the Jira issue? * Put link here, and add [ZEPPELIN-3882](https://issues.apache.org/jira/projects/ZEPPELIN/issues/ZEPPELIN-3882) ### How should this be tested? * Execute the unit tests ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no * Added support for Neo4j temporal and point types * fix checkstyle --- docs/interpreter/neo4j.md | 3 + neo4j/pom.xml | 6 +- .../graph/neo4j/Neo4jCypherInterpreter.java| 14 +++ .../graph/neo4j/Neo4jCypherInterpreterTest.java| 105 + 4 files changed, 108 insertions(+), 20 deletions(-) diff --git a/docs/interpreter/neo4j.md b/docs/interpreter/neo4j.md index 1b14127..eec9e07 100644 --- a/docs/interpreter/neo4j.md +++ b/docs/interpreter/neo4j.md @@ -26,6 +26,9 @@ limitations under the License. ## Overview [Neo4j](https://neo4j.com/product/) is a native graph database, designed to store and process graphs from bottom to top. +### Supported Version + +The Neo4j Interpreter supports all Neo4j versions since v3 via the official [Neo4j Java Driver](https://github.com/neo4j/neo4j-java-driver) ![Neo4j - Interpreter - Video]({{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/neo4j-interpreter-video.gif) diff --git a/neo4j/pom.xml b/neo4j/pom.xml index b8a89ad..cc39fdc 100644 --- a/neo4j/pom.xml +++ b/neo4j/pom.xml @@ -33,9 +33,9 @@ Zeppelin: Neo4j interpreter - 1.4.3 - 3.2.3 - 3.2.3 + 1.7.1 + 3.4.10 + 3.4.10 2.8.9 neo4j diff --git a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java index bcb9d7b..d7f8485 100644 --- a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java +++ b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java @@ -201,6 +201,20 @@ public class Neo4jCypherInterpreter extends Interpreter { value = val.asList(); } else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.MAP())) { value = val.asMap(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.POINT())) { + value = val.asPoint(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE())) { + value = val.asLocalDate(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.TIME())) { + value = val.asOffsetTime(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_TIME())) { + value = val.asLocalTime(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_DATE_TIME())) { + value = val.asLocalDateTime(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE_TIME())) { + value = val.asZonedDateTime(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DURATION())) { + value = val.asIsoDuration(); } } if (value instanceof Collection) { diff --git a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java index 24bd513..7940d5f 100644 --- a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java +++ b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java @@ -55,10 +55,15 @@ public class Neo4jCypherInterpreterTest { private static final String REL_KNOWS = "KNOWS"; private static final String CYPHER_FOREACH = - "FOREACH (x in range(1,1000) | CREATE (:%s{name: \"name\" + x, age: %s}))"; - private static final String CHPHER_UNWIND = "UNWIND range(1,1000) as x " -+ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(ran
[spark] branch master updated: [SPARK-26642][K8S] Add --num-executors option to spark-submit for Spark on K8S.
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6c18d8d [SPARK-26642][K8S] Add --num-executors option to spark-submit for Spark on K8S. 6c18d8d is described below commit 6c18d8d8079ac4d2d6dc7539601ab83fc5b51760 Author: Luca Canali AuthorDate: Sun Jan 20 12:43:34 2019 -0800 [SPARK-26642][K8S] Add --num-executors option to spark-submit for Spark on K8S. ## What changes were proposed in this pull request? This PR proposes to extend the spark-submit option --num-executors to be applicable to Spark on K8S too. It is motivated by convenience, for example when migrating jobs written for YARN to run on K8S. ## How was this patch tested? Manually tested on a K8S cluster. Author: Luca Canali Closes #23573 from LucaCanali/addNumExecutorsToK8s. --- core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 4 ++-- .../main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala index b403cc4..d5e17ff 100644 --- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala +++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala @@ -537,14 +537,14 @@ private[spark] class SparkSubmit extends Logging { // Yarn only OptionAssigner(args.queue, YARN, ALL_DEPLOY_MODES, confKey = "spark.yarn.queue"), - OptionAssigner(args.numExecutors, YARN, ALL_DEPLOY_MODES, -confKey = EXECUTOR_INSTANCES.key), OptionAssigner(args.pyFiles, YARN, ALL_DEPLOY_MODES, confKey = "spark.yarn.dist.pyFiles"), OptionAssigner(args.jars, YARN, ALL_DEPLOY_MODES, confKey = "spark.yarn.dist.jars"), OptionAssigner(args.files, YARN, ALL_DEPLOY_MODES, confKey = "spark.yarn.dist.files"), OptionAssigner(args.archives, YARN, ALL_DEPLOY_MODES, confKey = "spark.yarn.dist.archives"), // Other options + OptionAssigner(args.numExecutors, YARN | KUBERNETES, ALL_DEPLOY_MODES, +confKey = EXECUTOR_INSTANCES.key), OptionAssigner(args.executorCores, STANDALONE | YARN | KUBERNETES, ALL_DEPLOY_MODES, confKey = EXECUTOR_CORES.key), OptionAssigner(args.executorMemory, STANDALONE | MESOS | YARN | KUBERNETES, ALL_DEPLOY_MODES, diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala index f5e4c4a..9692d2a 100644 --- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala +++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala @@ -585,15 +585,15 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S | in standalone mode). | | Spark on YARN and Kubernetes only: +| --num-executors NUM Number of executors to launch (Default: 2). +| If dynamic allocation is enabled, the initial number of +| executors will be at least NUM. | --principal PRINCIPAL Principal to be used to login to KDC. | --keytab KEYTAB The full path to the file that contains the keytab for the | principal specified above. | | Spark on YARN only: | --queue QUEUE_NAME The YARN queue to submit to (Default: "default"). -| --num-executors NUM Number of executors to launch (Default: 2). -| If dynamic allocation is enabled, the initial number of -| executors will be at least NUM. | --archives ARCHIVES Comma separated list of archives to be extracted into the | working directory of each executor. """.stripMargin - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[zeppelin] branch master updated: [ZEPPELIN-3575] Add 'Copy Column Name' to table visualisation-table (#3254)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new 2b4920c [ZEPPELIN-3575] Add 'Copy Column Name' to table visualisation-table (#3254) 2b4920c is described below commit 2b4920c97484abd942acdd18dce3097eace7fc64 Author: Egor Klimov AuthorDate: Sat Jan 19 11:02:02 2019 +0300 [ZEPPELIN-3575] Add 'Copy Column Name' to table visualisation-table (#3254) ### What is this PR for? Add button to drop-down list in visualization table to copy column name. I know that column name could be obtained in a different way, for example, by auto-complete. Nevertheless I think this feature is comfortable. ### What type of PR is it? Feature ### What is the Jira issue? [ZEPPELIN-3575](https://issues.apache.org/jira/browse/ZEPPELIN-3575) ### How should this be tested? * Manually ### Screenshots (if appropriate) ![zp-20](https://user-images.githubusercontent.com/6136993/48303733-168ebc00-e51f-11e8-8317-008d59a6b55e.gif) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No * ZEPPELIN-3575 - Add 'Copy Column Name' button in visualization table * Move option --- .../app/visualization/builtins/visualization-table.js | 19 +++ 1 file changed, 19 insertions(+) diff --git a/zeppelin-web/src/app/visualization/builtins/visualization-table.js b/zeppelin-web/src/app/visualization/builtins/visualization-table.js index 179fd9a..4227258 100644 --- a/zeppelin-web/src/app/visualization/builtins/visualization-table.js +++ b/zeppelin-web/src/app/visualization/builtins/visualization-table.js @@ -230,6 +230,15 @@ export default class TableVisualization extends Visualization { gridOptions.columnDefs.map((colDef) => { colDef.menuItems = [ { + title: 'Copy Column Name', + action: function() { +self.copyStringToClipboard(this.context.col.displayName); + }, + active: function() { +return false; + }, +}, +{ title: 'Type: String', action: function() { self.updateColDefType(this.context.col.colDef, TableColumnType.STRING); @@ -515,4 +524,14 @@ export default class TableVisualization extends Visualization { }, }; } + + copyStringToClipboard(copyString) { +const strToClipboard = document.createElement('textarea'); +strToClipboard.value = copyString; +document.body.appendChild(strToClipboard); +strToClipboard.select(); +document.execCommand('copy'); +document.body.removeChild(strToClipboard); +return; + } }
[zeppelin] branch master updated: [ZEPPELIN-3944] Update Dockerfiles of spark_standalone and spark_yarn_cluster (#3282)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new 966a392 [ZEPPELIN-3944] Update Dockerfiles of spark_standalone and spark_yarn_cluster (#3282) 966a392 is described below commit 966a39240237edfd72b067afe867a244fae6474a Author: keineahnung2345 AuthorDate: Wed Jan 16 16:10:05 2019 +0800 [ZEPPELIN-3944] Update Dockerfiles of spark_standalone and spark_yarn_cluster (#3282) ### What is this PR for? Upgrade the Dockerfiles of spark_standalone and spark_yarn_cluster to CentOS7 and Spark 2.4.0. Java remains in version 7 since hadoop 2.x depends on Java7. ### What type of PR is it? Improvement ### What is the Jira issue? * [ZEPPELIN-3944](https://issues.apache.org/jira/browse/ZEPPELIN-3944) ### How should this be tested? * Follow the instructions here: [spark-standalone](https://zeppelin.apache.org/docs/0.8.0/setup/deployment/spark_cluster_mode.html#spark-standalone-mode) and [spark-yarn-cluster](https://zeppelin.apache.org/docs/0.8.0/setup/deployment/spark_cluster_mode.html#spark-on-yarn-mode) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No * Update spark_standalone Dockerfile * update yarn dockerfile * back to java 7 since hadoop 2.x depends on java 7 * back to java7 hadoop 2.x depends on java 7 * change the path to /xxx/.../jre-1.7.0-openjdk --- scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile | 4 ++-- .../docker/spark-cluster-managers/spark_standalone/Dockerfile | 10 +- .../spark-cluster-managers/spark_yarn_cluster/Dockerfile | 8 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile b/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile index 0eb26ca..b87838e 100644 --- a/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile +++ b/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile @@ -36,8 +36,8 @@ yum clean all # Remove old jdk RUN yum remove java; yum remove jdk -# install jdk8 -RUN yum install -y java-1.8.0-openjdk-devel +# install jdk7 +RUN yum install -y java-1.7.0-openjdk-devel ENV JAVA_HOME /usr/lib/jvm/java ENV PATH $PATH:$JAVA_HOME/bin diff --git a/scripts/docker/spark-cluster-managers/spark_standalone/Dockerfile b/scripts/docker/spark-cluster-managers/spark_standalone/Dockerfile index 8bf0f8d..e4fb780 100644 --- a/scripts/docker/spark-cluster-managers/spark_standalone/Dockerfile +++ b/scripts/docker/spark-cluster-managers/spark_standalone/Dockerfile @@ -12,10 +12,10 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -FROM centos:centos6 +FROM centos:centos7 -ENV SPARK_PROFILE 2.1 -ENV SPARK_VERSION 2.1.2 +ENV SPARK_PROFILE 2.4 +ENV SPARK_VERSION 2.4.0 ENV HADOOP_PROFILE 2.7 ENV SPARK_HOME /usr/local/spark @@ -33,13 +33,13 @@ yum clean all # Remove old jdk RUN yum remove java; yum remove jdk -# install jdk7 +# install jdk7 RUN yum install -y java-1.7.0-openjdk-devel ENV JAVA_HOME /usr/lib/jvm/java ENV PATH $PATH:$JAVA_HOME/bin # install spark -RUN curl -s http://apache.mirror.cdnetworks.com/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE.tgz | tar -xz -C /usr/local/ +RUN curl -s http://www.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE.tgz | tar -xz -C /usr/local/ RUN cd /usr/local && ln -s spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE spark # update boot script diff --git a/scripts/docker/spark-cluster-managers/spark_yarn_cluster/Dockerfile b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/Dockerfile index cbbda20..da3df1c 100644 --- a/scripts/docker/spark-cluster-managers/spark_yarn_cluster/Dockerfile +++ b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/Dockerfile @@ -12,10 +12,10 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -FROM centos:centos6 +FROM centos:centos7 -ENV SPARK_PROFILE 2.1 -ENV SPARK_VERSION 2.1.2 +ENV SPARK_PROFILE 2.4 +ENV SPARK_VERSION 2.4.0 ENV HADOOP_PROFILE 2.7 ENV HADOOP_VERSION 2.7.0 @@ -52,7 +52,7 @@ ENV HADOOP_MAPRED_HOME /usr/local/hadoop ENV HADOOP_YARN_HOME /usr/local/hadoop ENV HADOOP_CONF_DIR /usr/local/hadoop/etc/hadoop -RUN sed -i '/^export JAVA_HOME/ s:.*:export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64\nexport HADOOP_PREFIX=/usr/local/hadoop\nexport HAD
[spark] branch branch-2.3 updated: [SPARK-26120][TESTS][SS][SPARKR] Fix a streaming query leak in Structured Streaming R tests
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new 1979712 [SPARK-26120][TESTS][SS][SPARKR] Fix a streaming query leak in Structured Streaming R tests 1979712 is described below commit 19797124f1e169138258c8c113874ec6ffedbe3d Author: Shixiong Zhu AuthorDate: Wed Nov 21 09:31:12 2018 +0800 [SPARK-26120][TESTS][SS][SPARKR] Fix a streaming query leak in Structured Streaming R tests ## What changes were proposed in this pull request? Stop the streaming query in `Specify a schema by using a DDL-formatted string when reading` to avoid outputting annoying logs. ## How was this patch tested? Jenkins Closes #23089 from zsxwing/SPARK-26120. Authored-by: Shixiong Zhu Signed-off-by: hyukjinkwon (cherry picked from commit 4b7f7ef5007c2c8a5090f22c6e08927e9f9a407b) Signed-off-by: Felix Cheung --- R/pkg/tests/fulltests/test_streaming.R | 1 + 1 file changed, 1 insertion(+) diff --git a/R/pkg/tests/fulltests/test_streaming.R b/R/pkg/tests/fulltests/test_streaming.R index bfb1a04..6f0d2ae 100644 --- a/R/pkg/tests/fulltests/test_streaming.R +++ b/R/pkg/tests/fulltests/test_streaming.R @@ -127,6 +127,7 @@ test_that("Specify a schema by using a DDL-formatted string when reading", { expect_false(awaitTermination(q, 5 * 1000)) callJMethod(q@ssq, "processAllAvailable") expect_equal(head(sql("SELECT count(*) FROM people3"))[[1]], 3) + stopQuery(q) expect_error(read.stream(path = parquetPath, schema = "name stri"), "DataType stri is not supported.") - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: add checker note (#168)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 70a6071 add checker note (#168) 70a6071 is described below commit 70a60716fed85bfcfce4188b26faeb63ecbe2b79 Author: Felix Cheung AuthorDate: Sat Jan 12 20:07:14 2019 -0500 add checker note (#168) dd note to check checker after release --- release-process.md| 4 ++-- site/release-process.html | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/release-process.md b/release-process.md index 14c9c16..31d395c 100644 --- a/release-process.md +++ b/release-process.md @@ -130,7 +130,7 @@ that looks something like `[RESULT] [VOTE]...`. **THIS STEP IS IRREVERSIBLE so make sure you selected the correct staging repository. Once you move the artifacts into the release folder, they cannot be removed.** -After the vote passes, to upload the binaries to Apache mirrors, you move the binaries from dev directory (this should be where they are voted) to release directory. This "moving" is the only way you can add stuff to the actual release directory. +After the vote passes, to upload the binaries to Apache mirrors, you move the binaries from dev directory (this should be where they are voted) to release directory. This "moving" is the only way you can add stuff to the actual release directory. (Note: only PMC can move to release directory) ``` # Move the sub-directory in "dev" to the @@ -146,7 +146,7 @@ curl "https://dist.apache.org/repos/dist/dev/spark/KEYS"; > svn-spark/KEYS Verify that the resources are present in https://www.apache.org/dist/spark/";>https://www.apache.org/dist/spark/. It may take a while for them to be visible. This will be mirrored throughout the Apache network. -There are a few remaining steps. +Check the release checker result of the release at https://checker.apache.org/projs/spark.html";>https://checker.apache.org/projs/spark.html. For Maven Central Repository, you can Release from the https://repository.apache.org/";>Apache Nexus Repository Manager. This is already populated by the `release-build.sh publish-release` step. Log in, open Staging Repositories, find the one voted on (eg. orgapachespark-1257 for https://repository.apache.org/content/repositories/orgapachespark-1257/), select and click Release and confirm. If successful, it should show up under https://repository.apache.org/content/repositori [...] diff --git a/site/release-process.html b/site/release-process.html index 2a4b82b..65f8fb4 100644 --- a/site/release-process.html +++ b/site/release-process.html @@ -334,7 +334,7 @@ that looks something like [RESULT] [VOTE] THIS STEP IS IRREVERSIBLE so make sure you selected the correct staging repository. Once you move the artifacts into the release folder, they cannot be removed. -After the vote passes, to upload the binaries to Apache mirrors, you move the binaries from dev directory (this should be where they are voted) to release directory. This “moving” is the only way you can add stuff to the actual release directory. +After the vote passes, to upload the binaries to Apache mirrors, you move the binaries from dev directory (this should be where they are voted) to release directory. This “moving” is the only way you can add stuff to the actual release directory. (Note: only PMC can move to release directory) # Move the sub-directory in "dev" to the # corresponding directory in "release" @@ -349,7 +349,7 @@ curl "https://dist.apache.org/repos/dist/dev/spark/KEYS"; > svn-spark/KEYS Verify that the resources are present in https://www.apache.org/dist/spark/";>https://www.apache.org/dist/spark/. It may take a while for them to be visible. This will be mirrored throughout the Apache network. -There are a few remaining steps. +Check the release checker result of the release at https://checker.apache.org/projs/spark.html";>https://checker.apache.org/projs/spark.html. For Maven Central Repository, you can Release from the https://repository.apache.org/";>Apache Nexus Repository Manager. This is already populated by the release-build.sh publish-release step. Log in, open Staging Repositories, find the one voted on (eg. orgapachespark-1257 for https://repository.apache.org/content/repositories/orgapachespark-1257/), select and click Release and confirm. If successful, it should show up under https://repository.apache.org/cont [...] and the same under https://repository.apache.org/content/groups/maven-staging-group/org/apache/spark/spark-core_2.11/2.2.1/ (look for the correct release version). After some time this will be sync’d to https://search.maven.org/";>Maven Central automatically.
[spark] branch branch-2.3 updated: [SPARK-26010][R] fix vignette eval with Java 11
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new a9a1bc7 [SPARK-26010][R] fix vignette eval with Java 11 a9a1bc7 is described below commit a9a1bc78e619997007853be568a134ce3d1bbb14 Author: Felix Cheung AuthorDate: Mon Nov 12 19:03:30 2018 -0800 [SPARK-26010][R] fix vignette eval with Java 11 ## What changes were proposed in this pull request? changes in vignette only to disable eval ## How was this patch tested? Jenkins Author: Felix Cheung Closes #23007 from felixcheung/rjavavervig. (cherry picked from commit 88c82627267a9731b2438f0cc28dd656eb3dc834) Signed-off-by: Felix Cheung --- R/pkg/vignettes/sparkr-vignettes.Rmd | 14 ++ 1 file changed, 14 insertions(+) diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd b/R/pkg/vignettes/sparkr-vignettes.Rmd index d4713de..70970bd 100644 --- a/R/pkg/vignettes/sparkr-vignettes.Rmd +++ b/R/pkg/vignettes/sparkr-vignettes.Rmd @@ -57,6 +57,20 @@ First, let's load and attach the package. library(SparkR) ``` +```{r, include=FALSE} +# disable eval if java version not supported +override_eval <- tryCatch(!is.numeric(SparkR:::checkJavaVersion()), + error = function(e) { TRUE }, + warning = function(e) { TRUE }) + +if (override_eval) { + opts_hooks$set(eval = function(options) { +options$eval = FALSE +options + }) +} +``` + `SparkSession` is the entry point into SparkR which connects your R program to a Spark cluster. You can create a `SparkSession` using `sparkR.session` and pass in options such as the application name, any Spark packages depended on, etc. We use default settings in which it runs in local mode. It auto downloads Spark package in the background if no previous installation is found. For more details about setup, see [Spark Session](#SetupSparkSession). - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[zeppelin] branch master updated: [ZEPPELIN-3932] spark_mesos Dockerfile should be updated (#3279)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new 7bf79d8 [ZEPPELIN-3932] spark_mesos Dockerfile should be updated (#3279) 7bf79d8 is described below commit 7bf79d80f5beaaafa3c0e18a96654e43a20a41f9 Author: keineahnung2345 AuthorDate: Sat Jan 5 13:15:03 2019 +0800 [ZEPPELIN-3932] spark_mesos Dockerfile should be updated (#3279) ### What is this PR for? In the [spark_mesos example - Dockerfile](https://github.com/apache/zeppelin/blob/master/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile), there are some issues. 1. The original used SPARK_VERSION=2.1.2 is not available now, update it to 2.4.0 2. There is no package named libevent2-devel, it should be corrected to libevent-devel 3. Update from centos6 to centos7 4. Update from jdk7 to jdk8 In the [spark_mesos example - entrypoint.sh](https://github.com/apache/zeppelin/blob/master/scripts/docker/spark-cluster-managers/spark_mesos/entrypoint.sh): 1. Follow the instructions: [zeppelin on spark mesos mode - configure spark interpreter in zeppelin](https://zeppelin.apache.org/docs/0.8.0/setup/deployment/spark_cluster_mode.html#4-configure-spark-interpreter-in-zeppelin-1) add the two environment variables. 2. add --hostname flag to the command mesos-master. This solves the problem of "Failed to connect to xx.xx.xx.xx:5050"(screenshot). ### What type of PR is - Bug Fix - Improvement ### What is the Jira issue? * [ZEPPELIN-3932](https://issues.apache.org/jira/browse/ZEPPELIN-3932) ### How should this be tested? * Follow the instructions here: [spark-on-mesos-mode](https://zeppelin.apache.org/docs/0.8.0/setup/deployment/spark_cluster_mode.html#spark-on-mesos-mode) ### Screenshots (if appropriate) ![image](https://user-images.githubusercontent.com/18047300/50578759-fa513080-0e78-11e9-8459-3a2aa5881a2c.png) * update mesos Dockerfile 1. SPARK_VERSION 2.1.2 is not available anymore 2. centos cannot find libevent2-devel, but libevent-devel 3. update to centos7 and jdk8 * update entrypoint.sh according to tutorial 1. add environment variables MASTER and MESOS_NATIVE_JAVA_LIBRARY(from https://zeppelin.apache.org/docs/0.8.0/setup/deployment/spark_cluster_mode.html#4-configure-spark-interpreter-in-zeppelin-1) 2. add --hostname after mesos-master to solve the problem of "Failed to connect to xx.xx.xx.xx:5050" --- .../spark-cluster-managers/spark_mesos/Dockerfile | 26 +++--- .../spark_mesos/entrypoint.sh | 4 +++- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile b/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile index 0afda57..0eb26ca 100644 --- a/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile +++ b/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile @@ -12,10 +12,10 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -FROM centos:centos6 +FROM centos:centos7 -ENV SPARK_PROFILE 2.1 -ENV SPARK_VERSION 2.1.2 +ENV SPARK_PROFILE 2.4 +ENV SPARK_VERSION 2.4.0 ENV HADOOP_PROFILE 2.7 ENV HADOOP_VERSION 2.7.0 @@ -29,15 +29,15 @@ tar \ curl \ svn \ cyrus-sasl-md5 \ -libevent2-devel \ +libevent-devel \ && \ yum clean all # Remove old jdk RUN yum remove java; yum remove jdk -# install jdk7 -RUN yum install -y java-1.7.0-openjdk-devel +# install jdk8 +RUN yum install -y java-1.8.0-openjdk-devel ENV JAVA_HOME /usr/lib/jvm/java ENV PATH $PATH:$JAVA_HOME/bin @@ -45,14 +45,9 @@ ENV PATH $PATH:$JAVA_HOME/bin RUN curl -s http://www.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE.tgz | tar -xz -C /usr/local/ RUN cd /usr/local && ln -s spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE spark -# update boot script -COPY entrypoint.sh /etc/entrypoint.sh -RUN chown root.root /etc/entrypoint.sh -RUN chmod 700 /etc/entrypoint.sh - # install mesos -RUN wget http://repos.mesosphere.com/el/6/x86_64/RPMS/mesos-1.0.0-2.0.89.centos65.x86_64.rpm -RUN rpm -Uvh mesos-1.0.0-2.0.89.centos65.x86_64.rpm +RUN wget http://repos.mesosphere.com/el/7/x86_64/RPMS/mesos-1.7.0-2.0.1.el7.x86_64.rpm +RUN rpm -Uvh mesos-1.7.0-2.0.1.el7.x86_64.rpm #spark EXPOSE 8080 7077 7072 8081 8082 @@ -60,4 +55,9 @@ EXPOSE 8080 7077 7072 8081 8082 #mesos EXPOSE 5050 5051 +# update boot script +COPY entrypoint.sh /etc/entrypoint.sh +RUN chown root.root /etc/entrypoint.sh +RUN chmod 700 /etc/entrypoint.sh + ENTRYPOINT ["/etc/en
[zeppelin] branch master updated: [ZEPPELIN-3914] upgrade Flink to 1.7.1 (#3266)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new 13fd50e [ZEPPELIN-3914] upgrade Flink to 1.7.1 (#3266) 13fd50e is described below commit 13fd50ee535cd28b237cb4af740d0f893dbff3ae Author: Xue Yu <278006...@qq.com> AuthorDate: Tue Jan 1 09:22:42 2019 +0800 [ZEPPELIN-3914] upgrade Flink to 1.7.1 (#3266) ### What is this PR for? This PR is for upgrading current Flink to 1.7.1 ### What type of PR is it? [Improvement] ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3914 ### How should this be tested? *manual ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no --- docs/interpreter/flink.md | 2 +- flink/pom.xml | 2 +- .../apache/zeppelin/flink/FlinkSQLScalaInterpreter.scala| 2 +- .../org/apache/zeppelin/flink/FlinkScalaInterpreter.scala | 13 - 4 files changed, 7 insertions(+), 12 deletions(-) diff --git a/docs/interpreter/flink.md b/docs/interpreter/flink.md index 2cf3125..d3f2223 100644 --- a/docs/interpreter/flink.md +++ b/docs/interpreter/flink.md @@ -50,7 +50,7 @@ At the "Interpreters" menu, you have to create a new Flink interpreter and provi -For more information about Flink configuration, you can find it [here](https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html). +For more information about Flink configuration, you can find it [here](https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html). ## How to test it's working You can find an example of Flink usage in the Zeppelin Tutorial folder or try the following word count example, by using the [Zeppelin notebook](https://www.zeppelinhub.com/viewer/notebooks/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL05GTGFicy96ZXBwZWxpbi1ub3RlYm9va3MvbWFzdGVyL25vdGVib29rcy8yQVFFREs1UEMvbm90ZS5qc29u) from Till Rohrmann's presentation [Interactive data analysis with Apache Flink](http://www.slideshare.net/tillrohrmann/data-analysis-49806564) for Apache Flink Meetup. diff --git a/flink/pom.xml b/flink/pom.xml index 7a374f2..331e19c 100644 --- a/flink/pom.xml +++ b/flink/pom.xml @@ -36,7 +36,7 @@ flink -1.5.2 +1.7.1 2.3.7 2.0.1 2.11 diff --git a/flink/src/main/scala/org/apache/zeppelin/flink/FlinkSQLScalaInterpreter.scala b/flink/src/main/scala/org/apache/zeppelin/flink/FlinkSQLScalaInterpreter.scala index 1694a44..b2d8d16 100644 --- a/flink/src/main/scala/org/apache/zeppelin/flink/FlinkSQLScalaInterpreter.scala +++ b/flink/src/main/scala/org/apache/zeppelin/flink/FlinkSQLScalaInterpreter.scala @@ -30,7 +30,7 @@ class FlinkSQLScalaInterpreter(scalaInterpreter: FlinkScalaInterpreter, def interpret(code: String, context: InterpreterContext): InterpreterResult = { try { - val table: Table = this.btenv.sql(code) + val table: Table = this.btenv.sqlQuery(code) val result = z.showData(table) return new InterpreterResult(InterpreterResult.Code.SUCCESS, result) } catch { diff --git a/flink/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala b/flink/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala index 14f8959..1d8b27e 100644 --- a/flink/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala +++ b/flink/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala @@ -26,7 +26,7 @@ import org.apache.flink.api.scala.FlinkShell._ import org.apache.flink.api.scala.{ExecutionEnvironment, FlinkILoop} import org.apache.flink.client.program.ClusterClient import org.apache.flink.configuration.GlobalConfiguration -import org.apache.flink.runtime.minicluster.{MiniCluster, StandaloneMiniCluster} +import org.apache.flink.runtime.minicluster.MiniCluster import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment import org.apache.flink.table.api.TableEnvironment import org.apache.flink.table.api.scala.{BatchTableEnvironment, StreamTableEnvironment} @@ -45,8 +45,7 @@ class FlinkScalaInterpreter(val properties: Properties) { lazy val LOGGER: Logger = LoggerFactory.getLogger(getClass) private var flinkILoop: FlinkILoop = _ - private var cluster: Option[Either[Either[StandaloneMiniCluster, MiniCluster], -ClusterClient[_]]] = _ + private var cluster: Option[Either[MiniCluster, ClusterClient[_]]] = _ private var scalaCompleter: ScalaCompleter = _ private val interpreterOutput = new InterpreterOutputStream(LOGGER) @@ -68,8 +67,7 @@ class FlinkScalaInterpreter(val properti
[zeppelin] branch master updated: docs: fix punctuation (#3197)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new c20c830 docs: fix punctuation (#3197) c20c830 is described below commit c20c8304bdc9cce2edd931f0d58d6f40a8fed39a Author: Gerard de Melo AuthorDate: Mon Dec 31 00:07:19 2018 +0100 docs: fix punctuation (#3197) ### What is this PR for? Minor update to fix punctuation-related typos in docs/usage/interpreter/overview.md ### What type of PR is it? Documentation --- docs/usage/interpreter/overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/usage/interpreter/overview.md b/docs/usage/interpreter/overview.md index 6580a59..da59076 100644 --- a/docs/usage/interpreter/overview.md +++ b/docs/usage/interpreter/overview.md @@ -139,8 +139,8 @@ This approach works, but is not particularly convenient. `ConfInterpreter` can p `ConfInterpreter` is a generic interpreter that can be used by any interpreter. The input format should be the property file format. It can be used to make custom settings for any interpreter. However, `ConfInterpreter` needs to be run before that interpreter process is launched. When that interpreter process is launched is determined by the interpreter mode setting. -So users need to understand the ([interpreter mode setting ](../usage/interpreter/interpreter_bindings_mode.html) of Zeppelin and be aware of when the interpreter process is launched. E.g., if we set the Spark interpreter setting as isolated per note, then, under this setting, each note will launch one interpreter process. -In this scenario, users need to put `ConfInterpreter` as the first paragraph as in the below example. Otherwise, the customized setting cannot be applied (actually it would report ERROR) +So users need to understand the [interpreter mode setting](../usage/interpreter/interpreter_bindings_mode.html) of Zeppelin and be aware of when the interpreter process is launched. E.g., if we set the Spark interpreter setting as isolated per note, then, under this setting, each note will launch one interpreter process. +In this scenario, users need to put `ConfInterpreter` as the first paragraph as in the below example. Otherwise, the customized setting cannot be applied (actually it would report ERROR).
[incubator-pinot-site] 01/01: updating trademark
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch felixcheung-tm-1 in repository https://gitbox.apache.org/repos/asf/incubator-pinot-site.git commit d11cf4bcaf40f3613ce8231a1a76d9b9d8d5b9b5 Author: Felix Cheung AuthorDate: Fri Dec 28 14:34:25 2018 -0800 updating trademark as per https://www.apache.org/foundation/marks/pmcs#attributions --- index.html | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/index.html b/index.html index 89e2a7a..2a8a4c2 100644 --- a/index.html +++ b/index.html @@ -20,7 +20,7 @@ -Apache Pinot +Apache Pinot™ (Incubating) -Apache Pinot +Apache Pinot™ (Incubating) Pinot is a realtime distributed OLAP datastore, which is used at LinkedIn to deliver scalable real time analytics with low latency. -It can ingest data from offline data sources (such as Hadoop and flat files) as well as online sources (such as Kafka). +It can ingest data from offline data sources (such as Apache Hadoop and flat files) as well as online sources (such as Apache Kafka). Pinot is designed to scale horizontally. http://github.com/linkedin/pinot";>Download @@ -87,9 +87,9 @@ Apache Pinot is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that [...] -Copyright © 2017 http://www.apache.org/"; target="_blank">The Apache Software Foundation +Copyright © 2018 http://www.apache.org/"; target="_blank">The Apache Software Foundation -Apache, Apache Pinot (incubating), the Apache feather and the Pinot logo are trademarks of The Apache Software Foundation +Apache, Apache Pinot (incubating), the Apache feather and the Pinot logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot-site] branch felixcheung-tm-1 created (now d11cf4b)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a change to branch felixcheung-tm-1 in repository https://gitbox.apache.org/repos/asf/incubator-pinot-site.git. at d11cf4b updating trademark This branch includes the following new commits: new d11cf4b updating trademark The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[zeppelin] branch master updated: [ZEPPELIN-3864] Fix Travis tests
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new 6639025 [ZEPPELIN-3864] Fix Travis tests 6639025 is described below commit 663902518762cbad2fdc4efa3d552ad13effe4b0 Author: Savalek - Aleksey Savelenko <30798933+sava...@users.noreply.github.com> AuthorDate: Tue Dec 25 22:35:45 2018 +0300 [ZEPPELIN-3864] Fix Travis tests ### What is this PR for? Maven `frontend plugin v1.4` incorrectly processed `npm` messages with error. This PR update `frontend plugin` to 1.6 from 1.4. ### What type of PR is it? Bug Fix ### What is the Jira issue? [ZEPPELIN-3864](https://issues.apache.org/jira/browse/ZEPPELIN-3864), [ZP-70] ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no --- pom.xml| 2 +- zeppelin-web/e2e/searchBlock.spec.js | 2 ++ zeppelin-web/package.json | 12 zeppelin-web/pom.xml | 2 +- .../src/app/notebook/notebook.controller.js| 22 +++-- .../src/app/notebook/notebook.controller.test.js | 1 + .../note-create/note-create.controller.test.js | 10 +++--- .../components/note-list/note-list.factory.test.js | 36 +++--- zeppelin-web/webpack.config.js | 5 +-- 9 files changed, 49 insertions(+), 43 deletions(-) diff --git a/pom.xml b/pom.xml index 2465009..04a6ad8 100644 --- a/pom.xml +++ b/pom.xml @@ -102,7 +102,7 @@ v8.9.3 5.5.1 -1.4 +1.6 1.7.10 diff --git a/zeppelin-web/e2e/searchBlock.spec.js b/zeppelin-web/e2e/searchBlock.spec.js index 570673b..a146e47 100644 --- a/zeppelin-web/e2e/searchBlock.spec.js +++ b/zeppelin-web/e2e/searchBlock.spec.js @@ -20,6 +20,8 @@ describe('Search block e2e Test', function() { beforeEach(function() { browser.get('http://localhost:8080') +browser.sleep(500); +waitVisibility(element(by.linkText('Create new note'))) clickOn(element(by.linkText('Create new note'))) waitVisibility(element(by.id('noteCreateModal'))) clickAndWait(element(by.id('createNoteButton'))) diff --git a/zeppelin-web/package.json b/zeppelin-web/package.json index 69248c6..de83a62 100644 --- a/zeppelin-web/package.json +++ b/zeppelin-web/package.json @@ -22,7 +22,7 @@ "pree2e": "webdriver-manager update --gecko false --versions.chrome=2.35", "e2e": "protractor protractor.conf.js", "pretest": "npm rebuild phantomjs-prebuilt", -"test": "karma start karma.conf.js" +"karma-test": "karma start karma.conf.js" }, "dependencies": { "angular-ui-grid": "4.4.6", @@ -78,15 +78,15 @@ "html-webpack-plugin": "^3.2.0", "imports-loader": "^0.7.1", "istanbul-instrumenter-loader": "^0.2.0", -"jasmine-core": "^2.5.2", -"jasmine-spec-reporter": "^4.1.1", -"karma": "~1.3.0", +"jasmine-core": "^3.3.0", +"jasmine-spec-reporter": "^4.2.1", +"karma": "~3.1.3", "karma-coverage": "^1.1.2", -"karma-jasmine": "~1.0.2", +"karma-jasmine": "~2.0.1", "karma-phantomjs-launcher": "^1.0.4", "karma-sourcemap-loader": "^0.3.7", "karma-spec-reporter": "0.0.31", -"karma-webpack": "^1.8.1", +"karma-webpack": "^3.0.5", "load-grunt-tasks": "^0.4.0", "mini-css-extract-plugin": "^0.4.4", "ng-annotate-loader": "^0.2.0", diff --git a/zeppelin-web/pom.xml b/zeppelin-web/pom.xml index 498803b..7de15dc 100644 --- a/zeppelin-web/pom.xml +++ b/zeppelin-web/pom.xml @@ -111,7 +111,7 @@ test ${web.e2e.disabled} - run test + run karma-test diff --git a/zeppelin-web/src/app/notebook/notebook.controller.js b/zeppelin-web/src/app/notebook/notebook.controller.js index 2f1437d..426667d 100644 --- a/zeppelin-web/src/app/notebook/notebook.controller.js +++ b/zeppelin-web/src/app/notebook/notebook.controller.js @@ -1579,14 +1579,16 @@ function NotebookCtrl($scope, $route, $routeParams, $location, $rootScope, }); let content = document.getE
[zeppelin] branch branch-0.8 updated: [ZEPPELIN-3553] Fix URLs on "Multi-user Support" page
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch branch-0.8 in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/branch-0.8 by this push: new 2c2deb1 [ZEPPELIN-3553] Fix URLs on "Multi-user Support" page 2c2deb1 is described below commit 2c2deb1891b8700d3d9d68695891e09e3139a48d Author: mebelousov AuthorDate: Tue Jun 19 16:38:10 2018 +0300 [ZEPPELIN-3553] Fix URLs on "Multi-user Support" page ### What is this PR for? On page Setup > Multi-user Support http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/setup/basics/multi_user_support.html there are two urls: * Shiro Authentication - http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/setup/setup/security/shiro_authentication.html * Notebook Permission - http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/setup/setup/security/notebook_authorization.html Need to remove one of the "setup" ### What type of PR is it? Bug Fix ### What is the Jira issue? [ZEPPELIN-3553](https://issues.apache.org/jira/browse/ZEPPELIN-3553) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: mebelousov Closes #3035 from mebelousov/ZEPPELIN-3553 and squashes the following commits: 100a2af06 [mebelousov] ZEPPELIN-3553 Fix URLs for "Notebook Permission" and "Shiro Authentication" (cherry picked from commit efdc16d71c1c66e46ea67de21a4b1185f85c10f5) Signed-off-by: Felix Cheung --- docs/setup/basics/multi_user_support.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/setup/basics/multi_user_support.md b/docs/setup/basics/multi_user_support.md index 15d911c..e61b723 100644 --- a/docs/setup/basics/multi_user_support.md +++ b/docs/setup/basics/multi_user_support.md @@ -25,8 +25,8 @@ limitations under the License. This page describes about multi-user support. -- multiple users login / logout using [Shiro Authentication](../setup/security/shiro_authentication.html) -- managing [Notebook Permission](../setup/security/notebook_authorization.html) +- multiple users login / logout using [Shiro Authentication](../security/shiro_authentication.html) +- managing [Notebook Permission](../security/notebook_authorization.html) - how to setup [impersonation for interpreters](../../usage/interpreter/user_impersonation.html) - different contexts per user / note using [Interpreter Binding Mode](../../usage/interpreter/interpreter_binding_mode.html) - a paragraph in a notebook can be [Personalized](../../usage/other_features/personalized_mode.html)
[zeppelin] branch master updated: Revert "[ZEPPELIN-3882] Neo4jInterpreter - Support Point and Date Types" (#3270)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new 45fa3fd Revert "[ZEPPELIN-3882] Neo4jInterpreter - Support Point and Date Types" (#3270) 45fa3fd is described below commit 45fa3fdac0c79f78f6d0416e1bbfce281ef4e0df Author: Felix Cheung AuthorDate: Fri Dec 21 02:03:48 2018 -0500 Revert "[ZEPPELIN-3882] Neo4jInterpreter - Support Point and Date Types" (#3270) * Revert "fix checkstyle" This reverts commit 5542262568b3532fd11e2832d3168b386f35b3d6. * Revert "Added support for Neo4j temporal and point types" This reverts commit b98ccea98996eda183382a332379f3cfc428c0e9. * Revert "Added support for Neo4j temporal and point types" This reverts commit 0fbcddae980ff245eead964e8a5e0ee0f1a6. --- docs/interpreter/neo4j.md | 3 - neo4j/pom.xml | 6 +- .../graph/neo4j/Neo4jCypherInterpreter.java| 14 --- .../graph/neo4j/Neo4jCypherInterpreterTest.java| 105 - 4 files changed, 20 insertions(+), 108 deletions(-) diff --git a/docs/interpreter/neo4j.md b/docs/interpreter/neo4j.md index eec9e07..1b14127 100644 --- a/docs/interpreter/neo4j.md +++ b/docs/interpreter/neo4j.md @@ -26,9 +26,6 @@ limitations under the License. ## Overview [Neo4j](https://neo4j.com/product/) is a native graph database, designed to store and process graphs from bottom to top. -### Supported Version - -The Neo4j Interpreter supports all Neo4j versions since v3 via the official [Neo4j Java Driver](https://github.com/neo4j/neo4j-java-driver) ![Neo4j - Interpreter - Video]({{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/neo4j-interpreter-video.gif) diff --git a/neo4j/pom.xml b/neo4j/pom.xml index cc39fdc..b8a89ad 100644 --- a/neo4j/pom.xml +++ b/neo4j/pom.xml @@ -33,9 +33,9 @@ Zeppelin: Neo4j interpreter - 1.7.1 - 3.4.10 - 3.4.10 + 1.4.3 + 3.2.3 + 3.2.3 2.8.9 neo4j diff --git a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java index d7f8485..bcb9d7b 100644 --- a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java +++ b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java @@ -201,20 +201,6 @@ public class Neo4jCypherInterpreter extends Interpreter { value = val.asList(); } else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.MAP())) { value = val.asMap(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.POINT())) { - value = val.asPoint(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE())) { - value = val.asLocalDate(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.TIME())) { - value = val.asOffsetTime(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_TIME())) { - value = val.asLocalTime(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_DATE_TIME())) { - value = val.asLocalDateTime(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE_TIME())) { - value = val.asZonedDateTime(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DURATION())) { - value = val.asIsoDuration(); } } if (value instanceof Collection) { diff --git a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java index 7940d5f..24bd513 100644 --- a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java +++ b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java @@ -55,15 +55,10 @@ public class Neo4jCypherInterpreterTest { private static final String REL_KNOWS = "KNOWS"; private static final String CYPHER_FOREACH = - "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age: %s, " + - "address: point({ longitude: 56.7, latitude: 12.78, height: 8 }), " + - "birth: date('1984-04-04')}))"; - private static final String CHPHER_UNWIND = "UNWIND range(1,100) as x " -+ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 100) " + "FOREACH (x in range(1,1000) | CREATE (:%s{name: \"name\" + x, age: %s}))"; + private static final String CHPHER_UNWIND = "UNWIND range(1,1000) as x " ++ "MATCH (n), (m) WHERE id(n) = x AND
[zeppelin] branch revert-3244-update-neo4j-datatypes created (now 7ae0e41)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a change to branch revert-3244-update-neo4j-datatypes in repository https://gitbox.apache.org/repos/asf/zeppelin.git. at 7ae0e41 Revert "Added support for Neo4j temporal and point types" This branch includes the following new commits: new 4ddee85 Revert "fix checkstyle" new 0193be5 Revert "Added support for Neo4j temporal and point types" new 7ae0e41 Revert "Added support for Neo4j temporal and point types" The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[zeppelin] 01/03: Revert "fix checkstyle"
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch revert-3244-update-neo4j-datatypes in repository https://gitbox.apache.org/repos/asf/zeppelin.git commit 4ddee8579401c4bbf59f714a9972538684f555e5 Author: Felix Cheung AuthorDate: Thu Dec 20 23:03:14 2018 -0800 Revert "fix checkstyle" This reverts commit 5542262568b3532fd11e2832d3168b386f35b3d6. --- .../graph/neo4j/Neo4jCypherInterpreter.java| 14 --- .../graph/neo4j/Neo4jCypherInterpreterTest.java| 101 +++-- 2 files changed, 15 insertions(+), 100 deletions(-) diff --git a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java index d7f8485..bcb9d7b 100644 --- a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java +++ b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java @@ -201,20 +201,6 @@ public class Neo4jCypherInterpreter extends Interpreter { value = val.asList(); } else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.MAP())) { value = val.asMap(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.POINT())) { - value = val.asPoint(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE())) { - value = val.asLocalDate(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.TIME())) { - value = val.asOffsetTime(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_TIME())) { - value = val.asLocalTime(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_DATE_TIME())) { - value = val.asLocalDateTime(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE_TIME())) { - value = val.asZonedDateTime(); -} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DURATION())) { - value = val.asIsoDuration(); } } if (value instanceof Collection) { diff --git a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java index 7940d5f..f44cd77 100644 --- a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java +++ b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java @@ -55,15 +55,10 @@ public class Neo4jCypherInterpreterTest { private static final String REL_KNOWS = "KNOWS"; private static final String CYPHER_FOREACH = - "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age: %s, " + - "address: point({ longitude: 56.7, latitude: 12.78, height: 8 }), " + - "birth: date('1984-04-04')}))"; + "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age: %s, address: point({ longitude: 56.7, latitude: 12.78, height: 8 }), birth: date('1984-04-04')}))"; private static final String CHPHER_UNWIND = "UNWIND range(1,100) as x " + "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 100) " + "CREATE (n)-[:%s]->(m)"; - - private static final String TABLE_RESULT_PREFIX = "%table "; - private static final String NETWORK_RESULT_PREFIX = "%network "; @BeforeClass public static void setUpNeo4jServer() throws Exception { @@ -78,7 +73,7 @@ public class Neo4jCypherInterpreterTest { public static void tearDownNeo4jServer() throws Exception { server.close(); } - + @Before public void setUpZeppelin() { Properties p = new Properties(); @@ -88,7 +83,7 @@ public class Neo4jCypherInterpreterTest { interpreter = new Neo4jCypherInterpreter(p); context = InterpreterContext.builder() .setInterpreterOut(new InterpreterOutput(null)) -.build(); +.build();; } @After @@ -103,15 +98,14 @@ public class Neo4jCypherInterpreterTest { "return 'a' as colA, 'b' as colB, [1, 2, 3] as colC", context); assertEquals(Code.SUCCESS, result.code()); final String tableResult = "colA\tcolB\tcolC\n\"a\"\t\"b\"\t[1,2,3]\n"; -assertEquals(tableResult, result.toString().replace(TABLE_RESULT_PREFIX, StringUtils.EMPTY)); - +assertEquals(tableResult, result.toString().replace("%table ", StringUtils.EMPTY)); + result = interpreter.interpret( "return 'a' as colA, 'b' as colB, [{key: \"value\"}, {key: 1}] as colC", context); assertEquals(Code.SUCCESS, result.code()); final String tableResultWithMap = "colA\tcolB\tcolC\n\"a\"\t\"b\"\t[
[zeppelin] 02/03: Revert "Added support for Neo4j temporal and point types"
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch revert-3244-update-neo4j-datatypes in repository https://gitbox.apache.org/repos/asf/zeppelin.git commit 0193be5b312845f4ea4fed737a0441b001e1b403 Author: Felix Cheung AuthorDate: Thu Dec 20 23:03:14 2018 -0800 Revert "Added support for Neo4j temporal and point types" This reverts commit b98ccea98996eda183382a332379f3cfc428c0e9. --- docs/interpreter/neo4j.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/interpreter/neo4j.md b/docs/interpreter/neo4j.md index eec9e07..1b14127 100644 --- a/docs/interpreter/neo4j.md +++ b/docs/interpreter/neo4j.md @@ -26,9 +26,6 @@ limitations under the License. ## Overview [Neo4j](https://neo4j.com/product/) is a native graph database, designed to store and process graphs from bottom to top. -### Supported Version - -The Neo4j Interpreter supports all Neo4j versions since v3 via the official [Neo4j Java Driver](https://github.com/neo4j/neo4j-java-driver) ![Neo4j - Interpreter - Video]({{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/neo4j-interpreter-video.gif)
[zeppelin] 03/03: Revert "Added support for Neo4j temporal and point types"
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch revert-3244-update-neo4j-datatypes in repository https://gitbox.apache.org/repos/asf/zeppelin.git commit 7ae0e418a15540fffb2402391d122219e6819234 Author: Felix Cheung AuthorDate: Thu Dec 20 23:03:14 2018 -0800 Revert "Added support for Neo4j temporal and point types" This reverts commit 0fbcddae980ff245eead964e8a5e0ee0f1a6. --- neo4j/pom.xml | 6 +++--- .../org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/neo4j/pom.xml b/neo4j/pom.xml index cc39fdc..b8a89ad 100644 --- a/neo4j/pom.xml +++ b/neo4j/pom.xml @@ -33,9 +33,9 @@ Zeppelin: Neo4j interpreter - 1.7.1 - 3.4.10 - 3.4.10 + 1.4.3 + 3.2.3 + 3.2.3 2.8.9 neo4j diff --git a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java index f44cd77..24bd513 100644 --- a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java +++ b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java @@ -55,9 +55,9 @@ public class Neo4jCypherInterpreterTest { private static final String REL_KNOWS = "KNOWS"; private static final String CYPHER_FOREACH = - "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age: %s, address: point({ longitude: 56.7, latitude: 12.78, height: 8 }), birth: date('1984-04-04')}))"; - private static final String CHPHER_UNWIND = "UNWIND range(1,100) as x " -+ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 100) " + "FOREACH (x in range(1,1000) | CREATE (:%s{name: \"name\" + x, age: %s}))"; + private static final String CHPHER_UNWIND = "UNWIND range(1,1000) as x " ++ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 1000) " + "CREATE (n)-[:%s]->(m)"; @BeforeClass
[zeppelin] 02/03: Added support for Neo4j temporal and point types
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git commit b98ccea98996eda183382a332379f3cfc428c0e9 Author: Andrea Santurbano AuthorDate: Mon Nov 26 23:48:06 2018 +0100 Added support for Neo4j temporal and point types --- docs/interpreter/neo4j.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/interpreter/neo4j.md b/docs/interpreter/neo4j.md index 1b14127..eec9e07 100644 --- a/docs/interpreter/neo4j.md +++ b/docs/interpreter/neo4j.md @@ -26,6 +26,9 @@ limitations under the License. ## Overview [Neo4j](https://neo4j.com/product/) is a native graph database, designed to store and process graphs from bottom to top. +### Supported Version + +The Neo4j Interpreter supports all Neo4j versions since v3 via the official [Neo4j Java Driver](https://github.com/neo4j/neo4j-java-driver) ![Neo4j - Interpreter - Video]({{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/neo4j-interpreter-video.gif)
[zeppelin] branch master updated (a6f4562 -> 5542262)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git. from a6f4562 [ZEPPELIN-3902]upgrade jetty version new 0fbcd11 Added support for Neo4j temporal and point types new b98ccea Added support for Neo4j temporal and point types new 5542262 fix checkstyle The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/interpreter/neo4j.md | 3 + neo4j/pom.xml | 6 +- .../graph/neo4j/Neo4jCypherInterpreter.java| 14 +++ .../graph/neo4j/Neo4jCypherInterpreterTest.java| 105 + 4 files changed, 108 insertions(+), 20 deletions(-)
[zeppelin] 03/03: fix checkstyle
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git commit 5542262568b3532fd11e2832d3168b386f35b3d6 Author: Andrea Santurbano AuthorDate: Thu Nov 29 09:44:11 2018 +0100 fix checkstyle --- .../graph/neo4j/Neo4jCypherInterpreter.java| 14 +++ .../graph/neo4j/Neo4jCypherInterpreterTest.java| 101 ++--- 2 files changed, 100 insertions(+), 15 deletions(-) diff --git a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java index bcb9d7b..d7f8485 100644 --- a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java +++ b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java @@ -201,6 +201,20 @@ public class Neo4jCypherInterpreter extends Interpreter { value = val.asList(); } else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.MAP())) { value = val.asMap(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.POINT())) { + value = val.asPoint(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE())) { + value = val.asLocalDate(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.TIME())) { + value = val.asOffsetTime(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_TIME())) { + value = val.asLocalTime(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_DATE_TIME())) { + value = val.asLocalDateTime(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE_TIME())) { + value = val.asZonedDateTime(); +} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DURATION())) { + value = val.asIsoDuration(); } } if (value instanceof Collection) { diff --git a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java index f44cd77..7940d5f 100644 --- a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java +++ b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java @@ -55,10 +55,15 @@ public class Neo4jCypherInterpreterTest { private static final String REL_KNOWS = "KNOWS"; private static final String CYPHER_FOREACH = - "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age: %s, address: point({ longitude: 56.7, latitude: 12.78, height: 8 }), birth: date('1984-04-04')}))"; + "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age: %s, " + + "address: point({ longitude: 56.7, latitude: 12.78, height: 8 }), " + + "birth: date('1984-04-04')}))"; private static final String CHPHER_UNWIND = "UNWIND range(1,100) as x " + "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 100) " + "CREATE (n)-[:%s]->(m)"; + + private static final String TABLE_RESULT_PREFIX = "%table "; + private static final String NETWORK_RESULT_PREFIX = "%network "; @BeforeClass public static void setUpNeo4jServer() throws Exception { @@ -73,7 +78,7 @@ public class Neo4jCypherInterpreterTest { public static void tearDownNeo4jServer() throws Exception { server.close(); } - + @Before public void setUpZeppelin() { Properties p = new Properties(); @@ -83,7 +88,7 @@ public class Neo4jCypherInterpreterTest { interpreter = new Neo4jCypherInterpreter(p); context = InterpreterContext.builder() .setInterpreterOut(new InterpreterOutput(null)) -.build();; +.build(); } @After @@ -98,14 +103,15 @@ public class Neo4jCypherInterpreterTest { "return 'a' as colA, 'b' as colB, [1, 2, 3] as colC", context); assertEquals(Code.SUCCESS, result.code()); final String tableResult = "colA\tcolB\tcolC\n\"a\"\t\"b\"\t[1,2,3]\n"; -assertEquals(tableResult, result.toString().replace("%table ", StringUtils.EMPTY)); - +assertEquals(tableResult, result.toString().replace(TABLE_RESULT_PREFIX, StringUtils.EMPTY)); + result = interpreter.interpret( "return 'a' as colA, 'b' as colB, [{key: \"value\"}, {key: 1}] as colC", context); assertEquals(Code.SUCCESS, result.code()); final String tableResultWithMap = "colA\tcolB\tcolC\n\"a\"\t\"b\"\t[{\"key\":\"value\"},{\"key\":1}]\n"; -assertEquals(tableResultWithMap,
[zeppelin] 01/03: Added support for Neo4j temporal and point types
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git commit 0fbcddae980ff245eead964e8a5e0ee0f1a6 Author: Andrea Santurbano AuthorDate: Mon Nov 26 23:48:06 2018 +0100 Added support for Neo4j temporal and point types --- neo4j/pom.xml | 6 +++--- .../org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/neo4j/pom.xml b/neo4j/pom.xml index b8a89ad..cc39fdc 100644 --- a/neo4j/pom.xml +++ b/neo4j/pom.xml @@ -33,9 +33,9 @@ Zeppelin: Neo4j interpreter - 1.4.3 - 3.2.3 - 3.2.3 + 1.7.1 + 3.4.10 + 3.4.10 2.8.9 neo4j diff --git a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java index 24bd513..f44cd77 100644 --- a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java +++ b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java @@ -55,9 +55,9 @@ public class Neo4jCypherInterpreterTest { private static final String REL_KNOWS = "KNOWS"; private static final String CYPHER_FOREACH = - "FOREACH (x in range(1,1000) | CREATE (:%s{name: \"name\" + x, age: %s}))"; - private static final String CHPHER_UNWIND = "UNWIND range(1,1000) as x " -+ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 1000) " + "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age: %s, address: point({ longitude: 56.7, latitude: 12.78, height: 8 }), birth: date('1984-04-04')}))"; + private static final String CHPHER_UNWIND = "UNWIND range(1,100) as x " ++ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 100) " + "CREATE (n)-[:%s]->(m)"; @BeforeClass
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3258 in that case, can we make it build beam only when it has support for it. ie. build scala 2.10 - beam 2.0.0 build scala 2.11 - beam not included (but not failing) ---
[GitHub] zeppelin issue #3244: [ZEPPELIN-3882] Neo4jInterpreter - Support Point and D...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3244 merging if no more comment ---
[GitHub] zeppelin issue #3244: [ZEPPELIN-3882] Neo4jInterpreter - Support Point and D...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3244 I see it's green. not sure why the report above is not. ---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3258 > that's what i did in: ffca03c ah, that would make different versions of beam with scala-2.10 or scala-2.11. is there a scio version that works with scala-2.11? what if we upgrade the supported version of beam and scio in all profile? ---
[GitHub] zeppelin issue #3265: [ZEPPELIN-3910] IPython delete temp file and close str...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3265 can you check failed tests https://travis-ci.org/yx91490/zeppelin/builds/468316435 ---
[GitHub] zeppelin issue #3264: [ZEPPELIN-3908] Fix Date Sorting issue
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3264 please fix description - remove `First time? Setup Travis CI as described ` ... ---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3258 also we could upgrade beam interpreter to "require" beam 2.3 also (so it does build with scala-2.11) ---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3258 what about we don't build beam and scio if scala-2.11? ---
[GitHub] spark pull request #23252: [SPARK-26239] File-based secret key loading for S...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23252#discussion_r240022921 --- Diff: core/src/test/scala/org/apache/spark/SecurityManagerSuite.scala --- @@ -440,12 +473,27 @@ class SecurityManagerSuite extends SparkFunSuite with ResetSystemProperties { intercept[IllegalArgumentException] { mgr.getSecretKey() } + case FILE => +val secretFile = createTempSecretFile() +conf.set(AUTH_SECRET_FILE, secretFile.getAbsolutePath) +mgr.initializeAuth() +assert(encodeFileAsBase64(secretFile) === mgr.getSecretKey()) } } } ) } } + private def encodeFileAsBase64(secretFile: File) = { +Base64.getEncoder.encodeToString(Files.readAllBytes(secretFile.toPath)) + } + + private def createTempSecretFile(contents: String = "test-secret"): File = { +val secretDir = Utils.createTempDir("temp-secrets") +val secretFile = new File(secretDir, "temp-secret.txt") +Files.write(secretFile.toPath, contents.getBytes(StandardCharsets.UTF_8)) +secretFile --- End diff -- ah it's fine --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23224: [SPARK-26277][SQL][TEST] WholeStageCodegen metrics shoul...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23224 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23256: [SPARK-24207][R] follow-up PR for SPARK-24207 to fix cod...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23256 ideally, but really not for this PR --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23256: [SPARK-24207][R] follow-up PR for SPARK-24207 to ...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23256#discussion_r239997109 --- Diff: R/pkg/tests/fulltests/test_mllib_fpm.R --- @@ -84,19 +84,20 @@ test_that("spark.fpGrowth", { }) test_that("spark.prefixSpan", { -df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))), - list(list(list(1L), list(3L, 2L), list(1L, 2L))), - list(list(list(1L, 2L), list(5L))), - list(list(list(6L, schema = c("sequence")) -result1 <- spark.findFrequentSequentialPatterns(df, minSupport = 0.5, maxPatternLength = 5L, -maxLocalProjDBSize = 3200L) - -expected_result <- createDataFrame(list(list(list(list(1L)), 3L), -list(list(list(3L)), 2L), -list(list(list(2L)), 3L), -list(list(list(1L, 2L)), 3L), -list(list(list(1L), list(3L)), 2L)), -schema = c("sequence", "freq")) - }) + df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))), + list(list(list(1L), list(3L, 2L), list(1L, 2L))), + list(list(list(1L, 2L), list(5L))), + list(list(list(6L, +schema = c("sequence")) + result <- spark.findFrequentSequentialPatterns(df, minSupport = 0.5, maxPatternLength = 5L, + maxLocalProjDBSize = 3200L) + + expected_result <- createDataFrame(list(list(list(list(1L)), 3L), list(list(list(3L)), 2L), + list(list(list(2L)), 3L), list(list(list(1L, 2L)), 3L), + list(list(list(1L), list(3L)), 2L)), + schema = c("sequence", "freq")) + + expect_equivalent(expected_result, result) --- End diff -- this is an important fix.. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3235: [ZEPPELIN-3864] Fix Travis tests
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3235 Can you please fix the tests then. We canât merge anything that will cause all PRs to fail in Travis. Even when they âshouldâ be ---
[GitHub] spark issue #23218: [SPARK-26266][BUILD] Update to Scala 2.12.8
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23218 do we need to relnote jvm compatibility? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23252: [SPARK-26239] File-based secret key loading for S...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23252#discussion_r239705869 --- Diff: core/src/test/scala/org/apache/spark/SecurityManagerSuite.scala --- @@ -440,12 +473,27 @@ class SecurityManagerSuite extends SparkFunSuite with ResetSystemProperties { intercept[IllegalArgumentException] { mgr.getSecretKey() } + case FILE => +val secretFile = createTempSecretFile() +conf.set(AUTH_SECRET_FILE, secretFile.getAbsolutePath) +mgr.initializeAuth() +assert(encodeFileAsBase64(secretFile) === mgr.getSecretKey()) } } } ) } } + private def encodeFileAsBase64(secretFile: File) = { +Base64.getEncoder.encodeToString(Files.readAllBytes(secretFile.toPath)) + } + + private def createTempSecretFile(contents: String = "test-secret"): File = { +val secretDir = Utils.createTempDir("temp-secrets") +val secretFile = new File(secretDir, "temp-secret.txt") +Files.write(secretFile.toPath, contents.getBytes(StandardCharsets.UTF_8)) +secretFile --- End diff -- can this secret be recovered on disk or we trust tempDir ACL is sufficient? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3254: [ZEPPELIN-3575] Add 'Copy Column Name' to table visual...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3254 any comment on this? ---
[GitHub] zeppelin issue #3235: [ZEPPELIN-3864] Fix Travis tests
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3235 as I said, travis is failing with your fix? please check this test failed? https://travis-ci.org/TinkoffCreditSystems/zeppelin/jobs/458386760 ---
[GitHub] spark issue #22305: [SPARK-24561][SQL][Python] User-defined window aggregati...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22305 I can help if this looks good to @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23184#discussion_r238120855 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala --- @@ -225,4 +225,10 @@ private[sql] object SQLUtils extends Logging { } sparkSession.sessionState.catalog.listTables(db).map(_.table).toArray } + + def createArrayType(elementType: DataType): ArrayType = DataTypes.createArrayType(elementType) --- End diff -- yea, it's really minor, but really no one except you is working R APIs :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23184#discussion_r238120812 --- Diff: R/pkg/R/functions.R --- @@ -2254,40 +2255,48 @@ setMethod("date_format", signature(y = "Column", x = "character"), column(jc) }) +setClassUnion("characterOrstructTypeOrColumn", c("character", "structType", "Column")) --- End diff -- yes --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3245: [ZEPPELIN-3893] Bug Fix that clear paragraphs when exe...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3245 merging if no more comment ---
[GitHub] spark pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23072#discussion_r238087240 --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala --- @@ -64,4 +64,3 @@ object FPGrowthExample { spark.stop() } } -// scalastyle:on println --- End diff -- yes, println is not used --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3243: [ZEPPELIN-3886] Remove dependency on flatmap-stream 0....
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3243 could you check again? it seems travis tests are failing for other PRs because of event-stream now ---
[GitHub] zeppelin issue #3244: [ZEPPELIN-3882] Neo4jInterpreter - Support Point and D...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3244 looks like you are blocked by https://github.com/apache/zeppelin/pull/3243 https://api.travis-ci.org/v3/job/461718800/log.txt ---
[GitHub] spark issue #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23161 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
spark git commit: [SPARK-26189][R] Fix unionAll doc in SparkR
Repository: spark Updated Branches: refs/heads/master 28d337440 -> 2f6e88fec [SPARK-26189][R] Fix unionAll doc in SparkR ## What changes were proposed in this pull request? Fix unionAll doc in SparkR ## How was this patch tested? Manually ran test Author: Huaxin Gao Closes #23161 from huaxingao/spark-26189. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2f6e88fe Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2f6e88fe Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2f6e88fe Branch: refs/heads/master Commit: 2f6e88fecb455a02c4c08c41290e2f338e979543 Parents: 28d3374 Author: Huaxin Gao Authored: Fri Nov 30 23:14:05 2018 -0800 Committer: Felix Cheung Committed: Fri Nov 30 23:14:05 2018 -0800 -- R/pkg/R/DataFrame.R | 20 R/pkg/R/generics.R | 2 +- 2 files changed, 17 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/2f6e88fe/R/pkg/R/DataFrame.R -- diff --git a/R/pkg/R/DataFrame.R b/R/pkg/R/DataFrame.R index 745bb3e..24ed449 100644 --- a/R/pkg/R/DataFrame.R +++ b/R/pkg/R/DataFrame.R @@ -2730,13 +2730,25 @@ setMethod("union", dataFrame(unioned) }) -#' Return a new SparkDataFrame containing the union of rows +#' Return a new SparkDataFrame containing the union of rows. #' -#' This is an alias for `union`. +#' This is an alias for \code{union}. #' -#' @rdname union -#' @name unionAll +#' @param x a SparkDataFrame. +#' @param y a SparkDataFrame. +#' @return A SparkDataFrame containing the result of the unionAll operation. +#' @family SparkDataFrame functions #' @aliases unionAll,SparkDataFrame,SparkDataFrame-method +#' @rdname unionAll +#' @name unionAll +#' @seealso \link{union} +#' @examples +#'\dontrun{ +#' sparkR.session() +#' df1 <- read.json(path) +#' df2 <- read.json(path2) +#' unionAllDF <- unionAll(df1, df2) +#' } #' @note unionAll since 1.4.0 setMethod("unionAll", signature(x = "SparkDataFrame", y = "SparkDataFrame"), http://git-wip-us.apache.org/repos/asf/spark/blob/2f6e88fe/R/pkg/R/generics.R -- diff --git a/R/pkg/R/generics.R b/R/pkg/R/generics.R index 9d8c24c..eed7646 100644 --- a/R/pkg/R/generics.R +++ b/R/pkg/R/generics.R @@ -631,7 +631,7 @@ setGeneric("toRDD", function(x) { standardGeneric("toRDD") }) #' @rdname union setGeneric("union", function(x, y) { standardGeneric("union") }) -#' @rdname union +#' @rdname unionAll setGeneric("unionAll", function(x, y) { standardGeneric("unionAll") }) #' @rdname unionByName - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] spark issue #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23161 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23184#discussion_r238055143 --- Diff: R/pkg/R/functions.R --- @@ -2254,40 +2255,48 @@ setMethod("date_format", signature(y = "Column", x = "character"), column(jc) }) +setClassUnion("characterOrstructTypeOrColumn", c("character", "structType", "Column")) --- End diff -- we should probably try to pull all the setClassUnion in one place. (to avoid conflict or duplication) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23184#discussion_r238055087 --- Diff: R/pkg/R/functions.R --- @@ -2254,40 +2255,48 @@ setMethod("date_format", signature(y = "Column", x = "character"), column(jc) }) +setClassUnion("characterOrstructTypeOrColumn", c("character", "structType", "Column")) + #' @details #' \code{from_json}: Parses a column containing a JSON string into a Column of \code{structType} #' with the specified \code{schema} or array of \code{structType} if \code{as.json.array} is set #' to \code{TRUE}. If the string is unparseable, the Column will contain the value NA. #' #' @rdname column_collection_functions #' @param as.json.array indicating if input string is JSON array of objects or a single object. -#' @aliases from_json from_json,Column,characterOrstructType-method +#' @aliases from_json from_json,Column,characterOrstructTypeOrColumn-method #' @examples #' #' \dontrun{ #' df2 <- sql("SELECT named_struct('date', cast('2000-01-01' as date)) as d") #' df2 <- mutate(df2, d2 = to_json(df2$d, dateFormat = 'dd/MM/')) #' schema <- structType(structField("date", "string")) #' head(select(df2, from_json(df2$d2, schema, dateFormat = 'dd/MM/'))) - #' df2 <- sql("SELECT named_struct('name', 'Bob') as people") #' df2 <- mutate(df2, people_json = to_json(df2$people)) #' schema <- structType(structField("name", "string")) #' head(select(df2, from_json(df2$people_json, schema))) -#' head(select(df2, from_json(df2$people_json, "name STRING")))} +#' head(select(df2, from_json(df2$people_json, "name STRING"))) +#' head(select(df2, from_json(df2$people_json, schema_of_json(head(df2)$people_json} #' @note from_json since 2.2.0 -setMethod("from_json", signature(x = "Column", schema = "characterOrstructType"), +setMethod("from_json", signature(x = "Column", schema = "characterOrstructTypeOrColumn"), function(x, schema, as.json.array = FALSE, ...) { if (is.character(schema)) { - schema <- structType(schema) + jschema <- structType(schema)$jobj +} else if (class(schema) == "structType") { + jschema <- schema$jobj +} else { + jschema <- schema@jc } if (as.json.array) { - jschema <- callJStatic("org.apache.spark.sql.types.DataTypes", - "createArrayType", - schema$jobj) -} else { - jschema <- schema$jobj + # This case is R-specifically different. Unlike Scala and Python side, --- End diff -- what if `as.json.array` is TRUE but schema is also set? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23184#discussion_r238055126 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala --- @@ -225,4 +225,10 @@ private[sql] object SQLUtils extends Logging { } sparkSession.sessionState.catalog.listTables(db).map(_.table).toArray } + + def createArrayType(elementType: DataType): ArrayType = DataTypes.createArrayType(elementType) --- End diff -- as mentioned before, I kinda have to disagree with you here... I'd prefer less stuff in r/SQLUtils, and instead, call the scala/jvm method directly from R... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23184#discussion_r238055173 --- Diff: R/pkg/R/functions.R --- @@ -202,8 +202,9 @@ NULL #' \itemize{ #' \item \code{from_json}: a structType object to use as the schema to use #' when parsing the JSON string. Since Spark 2.3, the DDL-formatted string is -#' also supported for the schema. -#' \item \code{from_csv}: a DDL-formatted string +#' also supported for the schema. Since Spark 3.0, \code{schema_of_json} or +#' a string literal can also be accepted. --- End diff -- that's true...? they are the same? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22939: [SPARK-25446][R] Add schema_of_json() and schema_of_csv(...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22939 Error looks reasonable... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23161#discussion_r237383462 --- Diff: R/pkg/R/DataFrame.R --- @@ -2732,13 +2732,24 @@ setMethod("union", dataFrame(unioned) }) -#' Return a new SparkDataFrame containing the union of rows -#' -#' This is an alias for `union`. +#' Return a new SparkDataFrame containing the union of rows. +#' This is an alias for \code{union}. --- End diff -- actually, we do need a newline - L2735 is the title as this is a new page. could you build the roxygen2 doc to double check? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3243: [ZEPPELIN-3886] Remove dependency on flatmap-stream 0....
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3243 ouch, thanks for fixing this. @prabhjyotsingh @jongyoul @Leemoonsoo this is important. I checked and don't think we have released with this commit https://github.com/apache/zeppelin/commit/affccab6ab3577a815e02bb4d88aa1e90d973d16 https://issues.apache.org/jira/browse/ZEPPELIN-3781 ---
[GitHub] zeppelin issue #3242: [ZEPPELIN-3882] Neo4jInterpreter - Support Point and D...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3242 can you check the failed tests? ---
[GitHub] spark pull request #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23161#discussion_r236972877 --- Diff: R/pkg/R/DataFrame.R --- @@ -2732,14 +2732,24 @@ setMethod("union", dataFrame(unioned) }) -#' Return a new SparkDataFrame containing the union of rows -#' -#' This is an alias for `union`. +#' Return a new SparkDataFrame containing the union of rows. +#' This is an alias for \code{union}. #' -#' @rdname union -#' @name unionAll +#' @param x a SparkDataFrame. +#' @param y a SparkDataFrame. +#' @return A SparkDataFrame containing the result of the unionAll operation. +#' @family SparkDataFrame functions #' @aliases unionAll,SparkDataFrame,SparkDataFrame-method -#' @note unionAll since 1.4.0 +#' @rdname unionAll +#' @name unionAll --- End diff -- I'd add a `@seealso` to union --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23161#discussion_r236973169 --- Diff: R/pkg/R/DataFrame.R --- @@ -2732,14 +2732,24 @@ setMethod("union", dataFrame(unioned) }) -#' Return a new SparkDataFrame containing the union of rows -#' -#' This is an alias for `union`. +#' Return a new SparkDataFrame containing the union of rows. +#' This is an alias for \code{union}. #' -#' @rdname union -#' @name unionAll +#' @param x a SparkDataFrame. +#' @param y a SparkDataFrame. +#' @return A SparkDataFrame containing the result of the unionAll operation. +#' @family SparkDataFrame functions #' @aliases unionAll,SparkDataFrame,SparkDataFrame-method -#' @note unionAll since 1.4.0 +#' @rdname unionAll +#' @name unionAll +#' @examples +#'\dontrun{ +#' sparkR.session() +#' df1 <- read.json(path) +#' df2 <- read.json(path2) +#' unionAllDF <- unionAll(df1, df2) +#' } +#' @note unionAll since 3.0.0 --- End diff -- this should be `1.4.0` as above --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3242: [ZEPPELIN-3882] Neo4jInterpreter - Support Point and D...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3242 perhaps we should have a min version compatible in the interpreter doc? otherwise I think it's fine not to have tested version, supported version etc in doc since then we'd need to keep updating them ---
[GitHub] spark pull request #23025: [SPARK-26024][SQL]: Update documentation for repa...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23025#discussion_r236970732 --- Diff: R/pkg/R/DataFrame.R --- @@ -767,6 +767,14 @@ setMethod("repartition", #' using \code{spark.sql.shuffle.partitions} as number of partitions.} #'} #' +#' At least one partition-by expression must be specified. --- End diff -- 761 is significant also, but correct. essentially: 1. first line of the blob is the title (L760) 2. second text after "empty line" is the description (L762) 3. third after another "empty line" is the "detail note" which is stashed all the way to the bottom of the doc page so generally you want "important" part of the description on top and not in the "detail" section because it is easily missed. this is the most common pattern in this code base. there's another, where multiple function is doc together as a group, eg. collection sql function (in functions.R). other finer control is possible as well but not used today in this code base. similarly L829 is good, L831 is a bit fuzzy - I'd personally prefer without L831 to keep the whole text in the description section of the doc. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23072#discussion_r236771417 --- Diff: docs/ml-clustering.md --- @@ -265,3 +265,44 @@ Refer to the [R API docs](api/R/spark.gaussianMixture.html) for more details. + +## Power Iteration Clustering (PIC) + +Power Iteration Clustering (PIC) is a scalable graph clustering algorithm --- End diff -- could you open a separate PR with just this file (minus R) and FPGrowthExample.scala on branch-2.4? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22939: [SPARK-25446][R] Add schema_of_json() and schema_...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/22939#discussion_r236770223 --- Diff: R/pkg/R/functions.R --- @@ -2230,6 +2237,32 @@ setMethod("from_json", signature(x = "Column", schema = "characterOrstructType") column(jc) }) +#' @details +#' \code{schema_of_json}: Parses a JSON string and infers its schema in DDL format. +#' +#' @rdname column_collection_functions +#' @aliases schema_of_json schema_of_json,characterOrColumn-method +#' @examples +#' +#' \dontrun{ +#' json <- '{"name":"Bob"}' +#' df <- sql("SELECT * FROM range(1)") +#' head(select(df, schema_of_json(json)))} +#' @note schema_of_json since 3.0.0 +setMethod("schema_of_json", signature(x = "characterOrColumn"), + function(x, ...) { +if (class(x) == "character") { + col <- callJStatic("org.apache.spark.sql.functions", "lit", x) +} else { + col <- x@jc --- End diff -- maybe to think about the design of API in R and Scala and else where - what does it look like when the user passes in a column that is not a literal string? probably worthwhile to follow up separately. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3241: ZEPPELIN-3880: Code refactoring: remove class files us...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3241 tbh, this seems rather minor. I'd leave this for others to comment. ---
[GitHub] spark pull request #23145: [MINOR][Docs][WIP] Fix Typos
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23145#discussion_r236765511 --- Diff: docs/index.md --- @@ -67,7 +67,7 @@ Example applications are also provided in Python. For example, ./bin/spark-submit examples/src/main/python/pi.py 10 Spark also provides an experimental [R API](sparkr.html) since 1.4 (only DataFrames APIs included). --- End diff -- let's remove `experimental`, WDYT? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support fo...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23098#discussion_r236764795 --- Diff: R/pkg/R/sparkR.R --- @@ -269,7 +269,7 @@ sparkR.sparkContext <- function( #' sparkR.session("yarn-client", "SparkR", "/home/spark", #'list(spark.executor.memory="4g"), #'c("one.jar", "two.jar", "three.jar"), -#'c("com.databricks:spark-avro_2.11:2.0.1")) +#'c("com.databricks:spark-avro_2.12:2.0.1")) --- End diff -- yes, dummy name is completely fine with me. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23131: [SPARK-25908][SQL][FOLLOW-UP] Add back unionAll
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23131#discussion_r236763355 --- Diff: R/pkg/R/DataFrame.R --- @@ -2732,6 +2732,20 @@ setMethod("union", dataFrame(unioned) }) +#' Return a new SparkDataFrame containing the union of rows +#' +#' This is an alias for `union`. --- End diff -- also backtick doesn't format with roxygen2. this should be ``` This is an alias for \code{union}. ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23025: [SPARK-26024][SQL]: Update documentation for repa...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23025#discussion_r236762465 --- Diff: R/pkg/R/DataFrame.R --- @@ -767,6 +767,14 @@ setMethod("repartition", #' using \code{spark.sql.shuffle.partitions} as number of partitions.} #'} #' +#' At least one partition-by expression must be specified. --- End diff -- this won't be formatted correctly in R doc due to the fact that "empty line" is significant. L769 should be removed to ensure it is in description --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23131: [SPARK-25908][SQL][FOLLOW-UP] Add back unionAll
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23131#discussion_r236760822 --- Diff: R/pkg/R/DataFrame.R --- @@ -2732,6 +2732,20 @@ setMethod("union", dataFrame(unioned) }) +#' Return a new SparkDataFrame containing the union of rows +#' +#' This is an alias for `union`. --- End diff -- If the goal is for this to be like other *All, this should go into a separate doc page, plus seealso, example etc. The way this was written, as it was a deprecated function, this doc page merge with union - as it is, none of the text above will show up and also unionAll will not be listed in method index list. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23145: [MINOR][Docs] "a R interpreter" -> "an R interpre...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23145#discussion_r236546043 --- Diff: docs/index.md --- @@ -67,7 +67,7 @@ Example applications are also provided in Python. For example, ./bin/spark-submit examples/src/main/python/pi.py 10 Spark also provides an experimental [R API](sparkr.html) since 1.4 (only DataFrames APIs included). --- End diff -- tbh, I'm not sure this should be called "an experimental [R API]" --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3235: [ZEPPELIN-3864][ZP-68] Fix Travis tests
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3235 this test failed? https://travis-ci.org/TinkoffCreditSystems/zeppelin/jobs/458386760 ---
[GitHub] spark issue #23089: [SPARK-26120][TESTS][SS][SPARKR]Fix a streaming query le...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23089 Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22939: [SPARK-25446][R] Add schema_of_json() and schema_of_csv(...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22939 Sorry for the delay, will do another pass in 1 or 2 days --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23072#discussion_r234432181 --- Diff: R/pkg/R/mllib_clustering.R --- @@ -610,3 +616,57 @@ setMethod("write.ml", signature(object = "LDAModel", path = "character"), function(object, path, overwrite = FALSE) { write_internal(object, path, overwrite) }) + +#' PowerIterationClustering +#' +#' A scalable graph clustering algorithm. Users can call \code{spark.assignClusters} to +#' return a cluster assignment for each input vertex. +#' +# Run the PIC algorithm and returns a cluster assignment for each input vertex. +#' @param data A SparkDataFrame. +#' @param k The number of clusters to create. +#' @param initMode Param for the initialization algorithm. +#' @param maxIter Param for maximum number of iterations. +#' @param srcCol Param for the name of the input column for source vertex IDs. +#' @param dstCol Name of the input column for destination vertex IDs. +#' @param weightCol Param for weight column name. If this is not set or \code{NULL}, +#' we treat all instance weights as 1.0. +#' @param ... additional argument(s) passed to the method. +#' @return A dataset that contains columns of vertex id and the corresponding cluster for the id. +#' The schema of it will be: +#' \code{id: Long} +#' \code{cluster: Int} +#' @rdname spark.powerIterationClustering +#' @aliases assignClusters,PowerIterationClustering-method,SparkDataFrame-method +#' @examples +#' \dontrun{ +#' df <- createDataFrame(list(list(0L, 1L, 1.0), list(0L, 2L, 1.0), +#' list(1L, 2L, 1.0), list(3L, 4L, 1.0), +#' list(4L, 0L, 0.1)), schema = c("src", "dst", "weight")) +#' clusters <- spark.assignClusters(df, initMode="degree", weightCol="weight") +#' showDF(clusters) +#' } +#' @note spark.assignClusters(SparkDataFrame) since 3.0.0 +setMethod("spark.assignClusters", + signature(data = "SparkDataFrame"), + function(data, k = 2L, initMode = "random", maxIter = 20L, srcCol = "src", +dstCol = "dst", weightCol = NULL) { --- End diff -- I think we try to avoid srcCol dstCol in R (I think there are other R ml APIs like that) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23072#discussion_r234432019 --- Diff: R/pkg/R/mllib_clustering.R --- @@ -610,3 +616,57 @@ setMethod("write.ml", signature(object = "LDAModel", path = "character"), function(object, path, overwrite = FALSE) { write_internal(object, path, overwrite) }) + +#' PowerIterationClustering +#' +#' A scalable graph clustering algorithm. Users can call \code{spark.assignClusters} to +#' return a cluster assignment for each input vertex. +#' +# Run the PIC algorithm and returns a cluster assignment for each input vertex. +#' @param data A SparkDataFrame. +#' @param k The number of clusters to create. +#' @param initMode Param for the initialization algorithm. +#' @param maxIter Param for maximum number of iterations. +#' @param srcCol Param for the name of the input column for source vertex IDs. +#' @param dstCol Name of the input column for destination vertex IDs. +#' @param weightCol Param for weight column name. If this is not set or \code{NULL}, +#' we treat all instance weights as 1.0. +#' @param ... additional argument(s) passed to the method. +#' @return A dataset that contains columns of vertex id and the corresponding cluster for the id. +#' The schema of it will be: +#' \code{id: Long} +#' \code{cluster: Int} +#' @rdname spark.powerIterationClustering +#' @aliases assignClusters,PowerIterationClustering-method,SparkDataFrame-method +#' @examples +#' \dontrun{ +#' df <- createDataFrame(list(list(0L, 1L, 1.0), list(0L, 2L, 1.0), +#' list(1L, 2L, 1.0), list(3L, 4L, 1.0), +#' list(4L, 0L, 0.1)), schema = c("src", "dst", "weight")) +#' clusters <- spark.assignClusters(df, initMode="degree", weightCol="weight") +#' showDF(clusters) +#' } +#' @note spark.assignClusters(SparkDataFrame) since 3.0.0 +setMethod("spark.assignClusters", + signature(data = "SparkDataFrame"), + function(data, k = 2L, initMode = "random", maxIter = 20L, srcCol = "src", --- End diff -- set valid values for initMode and check for it - eg. https://github.com/apache/spark/pull/23072/files#diff-d9f92e07db6424e2527a7f9d7caa9013R355 and `match.arg(initMode)` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23072#discussion_r234432049 --- Diff: R/pkg/vignettes/sparkr-vignettes.Rmd --- @@ -968,6 +970,17 @@ predicted <- predict(model, df) head(predicted) ``` + Power Iteration Clustering + +Power Iteration Clustering (PIC) is a scalable graph clustering algorithm. `spark.assignClusters` method runs the PIC algorithm and returns a cluster assignment for each input vertex. + +```{r} +df <- createDataFrame(list(list(0L, 1L, 1.0), list(0L, 2L, 1.0), + list(1L, 2L, 1.0), list(3L, 4L, 1.0), + list(4L, 0L, 0.1)), schema = c("src", "dst", "weight")) +head(spark.assignClusters(df, initMode="degree", weightCol="weight")) --- End diff -- spacing: `initMode = "degree", weightCol = "weight"` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23073: [SPARK-26104] expose pci info to task scheduler
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23073#discussion_r234431864 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/ExecutorData.scala --- @@ -27,12 +27,14 @@ import org.apache.spark.rpc.{RpcAddress, RpcEndpointRef} * @param executorHost The hostname that this executor is running on * @param freeCores The current number of cores available for work on the executor * @param totalCores The total number of cores available to the executor + * @param pcis The external devices avaliable to the executor --- End diff -- available --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23073: [SPARK-26104] expose pci info to task scheduler
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23073 please put ^ comment into PR description (because comment is not included in commit message once the PR is merged) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 Yea there are some problem with some packages we depend on that are not installable from CRAN (eg too old) so it will be hard to a new version of R and new installation. So to clarify, depreciation as is we still test on R 3.1? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 Hey shane I donât think we are saying to test multiple R version at all. In fact quite the opposite, just the new(er) version at some point in the future. (We donât have a better solution for packages though. Thereâs another PR for R arrow package for example) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 I think it's easier to say unsupported if we are not testing it in jenkins or appveyer. I don't know if we any coverage at release for older R version anyway, so it's better to unsupported then deprecate. but agree maybe the way to do this is deprecate without updating R in jenkins --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23017: [WIP][SPARK-26015][K8S] Set a default UID for Spark on K...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23017 noted test issue. let's kick off test though --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23017: [WIP][SPARK-26015][K8S] Set a default UID for Spark on K...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23017 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23007: [SPARK-26010][R] fix vignette eval with Java 11
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23007 merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
spark git commit: [SPARK-26010][R] fix vignette eval with Java 11
Repository: spark Updated Branches: refs/heads/branch-2.4 3bc4c3330 -> 65e5b2659 [SPARK-26010][R] fix vignette eval with Java 11 ## What changes were proposed in this pull request? changes in vignette only to disable eval ## How was this patch tested? Jenkins Author: Felix Cheung Closes #23007 from felixcheung/rjavavervig. (cherry picked from commit 88c82627267a9731b2438f0cc28dd656eb3dc834) Signed-off-by: Felix Cheung Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/65e5b265 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/65e5b265 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/65e5b265 Branch: refs/heads/branch-2.4 Commit: 65e5b26590e66ac4220b5f60e11b7966746c8b08 Parents: 3bc4c33 Author: Felix Cheung Authored: Mon Nov 12 19:03:30 2018 -0800 Committer: Felix Cheung Committed: Mon Nov 12 19:03:56 2018 -0800 -- R/pkg/vignettes/sparkr-vignettes.Rmd | 14 ++ 1 file changed, 14 insertions(+) -- http://git-wip-us.apache.org/repos/asf/spark/blob/65e5b265/R/pkg/vignettes/sparkr-vignettes.Rmd -- diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd b/R/pkg/vignettes/sparkr-vignettes.Rmd index 090363c..b13f338 100644 --- a/R/pkg/vignettes/sparkr-vignettes.Rmd +++ b/R/pkg/vignettes/sparkr-vignettes.Rmd @@ -57,6 +57,20 @@ First, let's load and attach the package. library(SparkR) ``` +```{r, include=FALSE} +# disable eval if java version not supported +override_eval <- tryCatch(!is.numeric(SparkR:::checkJavaVersion()), + error = function(e) { TRUE }, + warning = function(e) { TRUE }) + +if (override_eval) { + opts_hooks$set(eval = function(options) { +options$eval = FALSE +options + }) +} +``` + `SparkSession` is the entry point into SparkR which connects your R program to a Spark cluster. You can create a `SparkSession` using `sparkR.session` and pass in options such as the application name, any Spark packages depended on, etc. We use default settings in which it runs in local mode. It auto downloads Spark package in the background if no previous installation is found. For more details about setup, see [Spark Session](#SetupSparkSession). - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] spark issue #22866: WIP [SPARK-12172][SPARKR] Remove internal-only RDD metho...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22866 thx, but DO NOT MERGE - there's some nasty bug I'm still investigating.. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
spark git commit: [SPARK-26010][R] fix vignette eval with Java 11
Repository: spark Updated Branches: refs/heads/master f9ff75653 -> 88c826272 [SPARK-26010][R] fix vignette eval with Java 11 ## What changes were proposed in this pull request? changes in vignette only to disable eval ## How was this patch tested? Jenkins Author: Felix Cheung Closes #23007 from felixcheung/rjavavervig. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/88c82627 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/88c82627 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/88c82627 Branch: refs/heads/master Commit: 88c82627267a9731b2438f0cc28dd656eb3dc834 Parents: f9ff756 Author: Felix Cheung Authored: Mon Nov 12 19:03:30 2018 -0800 Committer: Felix Cheung Committed: Mon Nov 12 19:03:30 2018 -0800 -- R/pkg/vignettes/sparkr-vignettes.Rmd | 14 ++ 1 file changed, 14 insertions(+) -- http://git-wip-us.apache.org/repos/asf/spark/blob/88c82627/R/pkg/vignettes/sparkr-vignettes.Rmd -- diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd b/R/pkg/vignettes/sparkr-vignettes.Rmd index 7d924ef..f80b45b 100644 --- a/R/pkg/vignettes/sparkr-vignettes.Rmd +++ b/R/pkg/vignettes/sparkr-vignettes.Rmd @@ -57,6 +57,20 @@ First, let's load and attach the package. library(SparkR) ``` +```{r, include=FALSE} +# disable eval if java version not supported +override_eval <- tryCatch(!is.numeric(SparkR:::checkJavaVersion()), + error = function(e) { TRUE }, + warning = function(e) { TRUE }) + +if (override_eval) { + opts_hooks$set(eval = function(options) { +options$eval = FALSE +options + }) +} +``` + `SparkSession` is the entry point into SparkR which connects your R program to a Spark cluster. You can create a `SparkSession` using `sparkR.session` and pass in options such as the application name, any Spark packages depended on, etc. We use default settings in which it runs in local mode. It auto downloads Spark package in the background if no previous installation is found. For more details about setup, see [Spark Session](#SetupSparkSession). - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org