[GitHub] [zeppelin] zjffdu commented on a change in pull request #3320: [ZEPPELIN-3977]. Create shell script for converting old note file to new file
zjffdu commented on a change in pull request #3320: [ZEPPELIN-3977]. Create
shell script for converting old note file to new file
URL: https://github.com/apache/zeppelin/pull/3320#discussion_r265439934
##
File path:
zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/NotebookRepoSync.java
##
@@ -62,59 +63,84 @@ public void init(ZeppelinConfiguration conf) throws
IOException {
oneWaySync = conf.getBoolean(ConfVars.ZEPPELIN_NOTEBOOK_ONE_WAY_SYNC);
String allStorageClassNames = conf.getNotebookStorageClass().trim();
if (allStorageClassNames.isEmpty()) {
- allStorageClassNames = defaultStorage;
+ allStorageClassNames = DEFAULT_STORAGE;
LOGGER.warn("Empty ZEPPELIN_NOTEBOOK_STORAGE conf parameter, using
default {}",
- defaultStorage);
+ DEFAULT_STORAGE);
}
String[] storageClassNames = allStorageClassNames.split(",");
if (storageClassNames.length > getMaxRepoNum()) {
LOGGER.warn("Unsupported number {} of storage classes in
ZEPPELIN_NOTEBOOK_STORAGE : {}\n" +
"first {} will be used", storageClassNames.length,
allStorageClassNames, getMaxRepoNum());
}
+// init the underlying NotebookRepo
for (int i = 0; i < Math.min(storageClassNames.length, getMaxRepoNum());
i++) {
NotebookRepo notebookRepo =
PluginManager.get().loadNotebookRepo(storageClassNames[i].trim());
if (notebookRepo != null) {
notebookRepo.init(conf);
repos.add(notebookRepo);
}
}
+
// couldn't initialize any storage, use default
if (getRepoCount() == 0) {
- LOGGER.info("No storage could be initialized, using default {} storage",
defaultStorage);
- NotebookRepo defaultNotebookRepo =
PluginManager.get().loadNotebookRepo(defaultStorage);
+ LOGGER.info("No storage could be initialized, using default {} storage",
DEFAULT_STORAGE);
+ NotebookRepo defaultNotebookRepo =
PluginManager.get().loadNotebookRepo(DEFAULT_STORAGE);
defaultNotebookRepo.init(conf);
repos.add(defaultNotebookRepo);
}
+// sync for anonymous mode on start
+if (getRepoCount() > 1 &&
conf.getBoolean(ConfVars.ZEPPELIN_ANONYMOUS_ALLOWED)) {
+ try {
+sync(AuthenticationInfo.ANONYMOUS);
+ } catch (IOException e) {
+LOGGER.error("Couldn't sync anonymous mode on start ", e);
+ }
+}
+ }
+
+ // Zeppelin change its note file name structure in 0.9.0, this is called
when upgrading
+ // from 0.9.0 before to 0.9.0 after
+ public void convertNoteFiles(ZeppelinConfiguration conf, boolean deleteOld)
throws IOException {
// convert old note file (noteId/note.json) to new note file
(note_name_note_id.zpln)
-boolean convertToNew =
conf.getBoolean(ConfVars.ZEPPELIN_NOTEBOOK_NEW_FORMAT_CONVERT);
-boolean deleteOld =
conf.getBoolean(ConfVars.ZEPPELIN_NOTEBOOK_NEW_FORMAT_DELETE_OLD);
-if (convertToNew) {
- NotebookRepo newNotebookRepo = repos.get(0);
+for (int i = 0; i < repos.size(); ++i) {
+ NotebookRepo newNotebookRepo = repos.get(i);
OldNotebookRepo oldNotebookRepo =
-
PluginManager.get().loadOldNotebookRepo(newNotebookRepo.getClass().getCanonicalName());
+
PluginManager.get().loadOldNotebookRepo(newNotebookRepo.getClass().getCanonicalName());
oldNotebookRepo.init(conf);
List oldNotesInfo =
oldNotebookRepo.list(AuthenticationInfo.ANONYMOUS);
LOGGER.info("Convert old note file to new style, note count: " +
oldNotesInfo.size());
+ LOGGER.info("Delete old note: " + deleteOld);
Review comment:
It must be old format, because OldNotebookRepo can only read old format note
file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3320: [ZEPPELIN-3977]. Create shell script for converting old note file to new file
zjffdu commented on a change in pull request #3320: [ZEPPELIN-3977]. Create shell script for converting old note file to new file URL: https://github.com/apache/zeppelin/pull/3320#discussion_r265440505 ## File path: docs/setup/operation/upgrading.md ## @@ -35,6 +35,12 @@ So, copying `notebook` and `conf` directory should be enough. ## Migration Guide +### Upgrading from Zeppelin 0.8 to 0.9 + + - From 0.9, we change the notes file name structure ([ZEPPELIN-2619](https://issues.apache.org/jira/browse/ZEPPELIN-2619)). So when you upgrading zeppelin to 0.9, you need to upgrade note file. Here's steps you need to follow: Review comment: I have added the version in note file and also update the commit message to include version. ``` commit b5ed201cab6cab13eed59cc1330b29937ae55889 (HEAD -> master) Author: Jeff Zhang Date: Thu Mar 14 15:03:58 2019 +0800 Upgrade note 'Untitled Note 1' to 0.9.0-SNAPSHOT commit f4d08fe5043eb6e905dea5ade45b6ccd2458e8c1 Author: Jeff Zhang Date: Thu Mar 14 15:03:58 2019 +0800 Upgrade note 'Basic Features (Spark)' to 0.9.0-SNAPSHOT ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter
liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265515902
##
File path:
submarine/src/main/java/org/apache/zeppelin/submarine/commons/SubmarineConstants.java
##
@@ -0,0 +1,119 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine.commons;
+
+/*
+ * NOTE: use lowercase + "_" for the option name
+ */
+public class SubmarineConstants {
+ // Docker container Environmental variable at `submarine-job-run-tf.jinja`
+ // and `/bin/interpreter.sh`
+ public static final String DOCKER_HADOOP_HDFS_HOME =
"DOCKER_HADOOP_HDFS_HOME";
+ public static final String DOCKER_JAVA_HOME= "DOCKER_JAVA_HOME";
+ public static final String DOCKER_CONTAINER_TIME_ZONE =
"DOCKER_CONTAINER_TIME_ZONE";
+ public static final String INTERPRETER_LAUNCH_MODE =
"INTERPRETER_LAUNCH_MODE";
+
+ // interpreter.sh Environmental variable
+ public static final String SUBMARINE_HADOOP_HOME = "SUBMARINE_HADOOP_HOME";
+ public static final String HADOOP_YARN_SUBMARINE_JAR =
"HADOOP_YARN_SUBMARINE_JAR";
+ public static final String SUBMARINE_INTERPRETER_DOCKER_IMAGE
+ = "SUBMARINE_INTERPRETER_DOCKER_IMAGE";
+
+ public static final String ZEPPELIN_SUBMARINE_AUTH_TYPE =
"zeppelin.submarine.auth.type";
+ public static final String SUBMARINE_HADOOP_CONF_DIR =
"SUBMARINE_HADOOP_CONF_DIR";
+ public static final String SUBMARINE_HADOOP_KEYTAB=
"SUBMARINE_HADOOP_KEYTAB";
+ public static final String SUBMARINE_HADOOP_PRINCIPAL =
"SUBMARINE_HADOOP_PRINCIPAL";
+ public static final String SUBMARINE_HADOOP_KRB5_CONF =
"submarine.hadoop.krb5.conf";
+
+ public static final String JOB_NAME = "JOB_NAME";
+ public static final String CLEAN_CHECKPOINT = "CLEAN_CHECKPOINT";
+ public static final String INPUT_PATH = "INPUT_PATH";
+ public static final String CHECKPOINT_PATH = "CHECKPOINT_PATH";
+ public static final String PS_LAUNCH_CMD = "PS_LAUNCH_CMD";
+ public static final String WORKER_LAUNCH_CMD = "WORKER_LAUNCH_CMD";
+ public static final String MACHINELEARING_DISTRIBUTED_ENABLE
+ = "machinelearing.distributed.enable";
+
+ public static final String ZEPPELIN_INTERPRETER_RPC_PORTRANGE
+ = "zeppelin.interpreter.rpc.portRange";
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter
liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265515861
##
File path:
submarine/src/main/java/org/apache/zeppelin/submarine/commons/SubmarineConstants.java
##
@@ -0,0 +1,119 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine.commons;
+
+/*
+ * NOTE: use lowercase + "_" for the option name
+ */
+public class SubmarineConstants {
+ // Docker container Environmental variable at `submarine-job-run-tf.jinja`
+ // and `/bin/interpreter.sh`
+ public static final String DOCKER_HADOOP_HDFS_HOME =
"DOCKER_HADOOP_HDFS_HOME";
+ public static final String DOCKER_JAVA_HOME= "DOCKER_JAVA_HOME";
+ public static final String DOCKER_CONTAINER_TIME_ZONE =
"DOCKER_CONTAINER_TIME_ZONE";
+ public static final String INTERPRETER_LAUNCH_MODE =
"INTERPRETER_LAUNCH_MODE";
+
+ // interpreter.sh Environmental variable
+ public static final String SUBMARINE_HADOOP_HOME = "SUBMARINE_HADOOP_HOME";
+ public static final String HADOOP_YARN_SUBMARINE_JAR =
"HADOOP_YARN_SUBMARINE_JAR";
+ public static final String SUBMARINE_INTERPRETER_DOCKER_IMAGE
+ = "SUBMARINE_INTERPRETER_DOCKER_IMAGE";
+
+ public static final String ZEPPELIN_SUBMARINE_AUTH_TYPE =
"zeppelin.submarine.auth.type";
+ public static final String SUBMARINE_HADOOP_CONF_DIR =
"SUBMARINE_HADOOP_CONF_DIR";
+ public static final String SUBMARINE_HADOOP_KEYTAB=
"SUBMARINE_HADOOP_KEYTAB";
+ public static final String SUBMARINE_HADOOP_PRINCIPAL =
"SUBMARINE_HADOOP_PRINCIPAL";
+ public static final String SUBMARINE_HADOOP_KRB5_CONF =
"submarine.hadoop.krb5.conf";
+
+ public static final String JOB_NAME = "JOB_NAME";
+ public static final String CLEAN_CHECKPOINT = "CLEAN_CHECKPOINT";
+ public static final String INPUT_PATH = "INPUT_PATH";
+ public static final String CHECKPOINT_PATH = "CHECKPOINT_PATH";
+ public static final String PS_LAUNCH_CMD = "PS_LAUNCH_CMD";
+ public static final String WORKER_LAUNCH_CMD = "WORKER_LAUNCH_CMD";
+ public static final String MACHINELEARING_DISTRIBUTED_ENABLE
+ = "machinelearing.distributed.enable";
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (ZEPPELIN-4069) Error saving dataframe into Hive (Cloudera CDH 6.1.1)
Hayk Zargaryan created ZEPPELIN-4069: Summary: Error saving dataframe into Hive (Cloudera CDH 6.1.1) Key: ZEPPELIN-4069 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4069 Project: Zeppelin Issue Type: Bug Affects Versions: 0.8.1 Reporter: Hayk Zargaryan Attachments: image-2019-03-14-23-40-10-296.png I can easily read Hive tables but when I try to save dataframe into Hive I receive the below error. After that I have to restart spark interpreter because the error appears every time accessing Hive !image-2019-03-14-23-40-10-296.png|width=974,height=504! Seems both Cloudera and Zeppelin use the same version of libfb303 jar $ md5sum /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/jars/libfb303-0.9.3.jar 5e1c646346ecf2750a1b8b6cb2aa1c4f /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/jars/libfb303-0.9.3.jar $ md5sum /opt/zeppelin-0.8.1/local-repo/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar 5e1c646346ecf2750a1b8b6cb2aa1c4f /opt/zeppelin-0.8.1/local-repo/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZEPPELIN-4070) "Command + /" comment shortcut does not consistently recognize interpreter with pyspark
Peter Varshavsky created ZEPPELIN-4070: -- Summary: "Command + /" comment shortcut does not consistently recognize interpreter with pyspark Key: ZEPPELIN-4070 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4070 Project: Zeppelin Issue Type: Bug Components: GUI Affects Versions: 0.7.3 Reporter: Peter Varshavsky When a paragraph uses `%pyspark` interpreter, the commenting shortcut "Command + /" sometimes produces the expected `#` comment symbol but sometimes produces `//`. This behavior is consistent within a paragraph, but might be inconsistent between paragraphs, e.g. first paragraph might do `//` and second might do `#`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZEPPELIN-4071) JDBC interpreter build failed
Tiger068 created ZEPPELIN-4071:
--
Summary: JDBC interpreter build failed
Key: ZEPPELIN-4071
URL: https://issues.apache.org/jira/browse/ZEPPELIN-4071
Project: Zeppelin
Issue Type: Bug
Components: build
Affects Versions: 0.8.1
Reporter: Tiger068
Fix For: 0.8.2
maven build zeppelin, module JDBC interpreters build failed
maven error log:
{code:java}
[WARNING] The requested profile "ossrh" could not be activated because it does
not exist. [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project zeppelin-jdbc: Compilation failure:
Compilation failure: [ERROR]
/Volumes/LaCie/codespace/zeppelin-github/jdbc/src/test/java/org/apache/zeppelin/jdbc/SqlCompleterTest.java:[35,30]
程序包com.google.common.base不存在 [ERROR]
/Volumes/LaCie/codespace/zeppelin-github/jdbc/src/test/java/org/apache/zeppelin/jdbc/SqlCompleterTest.java:[38,40]
程序包com.google.common.collect不存在 [ERROR]
/Volumes/LaCie/codespace/zeppelin-github/jdbc/src/test/java/org/apache/zeppelin/jdbc/SqlCompleterTest.java:[38,1]
仅从类和接口静态导入 [ERROR] -> [Help 1] {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [zeppelin] felixcheung commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter
felixcheung commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265849340 ## File path: scripts/docker/submarine/submarine-interpreter-cpu-1.0.0/Dockerfile ## @@ -0,0 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM zeppelin-tensorflow_1.10-hadoop_3.1.2-cpu:1.0.0 Review comment: hmm, it looks to be uncommon to put version number in image name? https://hub.docker.com/u/apache?page=2 other than that I don't have any good/useful suggestion. perhaps the latest you have is fine This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] felixcheung commented on a change in pull request #3320: [ZEPPELIN-3977]. Create shell script for converting old note file to new file
felixcheung commented on a change in pull request #3320: [ZEPPELIN-3977]. Create shell script for converting old note file to new file URL: https://github.com/apache/zeppelin/pull/3320#discussion_r265848679 ## File path: docs/setup/operation/upgrading.md ## @@ -35,6 +35,12 @@ So, copying `notebook` and `conf` directory should be enough. ## Migration Guide +### Upgrading from Zeppelin 0.8 to 0.9 + + - From 0.9, we change the notes file name structure ([ZEPPELIN-2619](https://issues.apache.org/jira/browse/ZEPPELIN-2619)). So when you upgrading zeppelin to 0.9, you need to upgrade note file. Here's steps you need to follow: Review comment: ok This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] felixcheung commented on a change in pull request #3320: [ZEPPELIN-3977]. Create shell script for converting old note file to new file
felixcheung commented on a change in pull request #3320: [ZEPPELIN-3977].
Create shell script for converting old note file to new file
URL: https://github.com/apache/zeppelin/pull/3320#discussion_r265848705
##
File path:
zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/NotebookRepoSync.java
##
@@ -62,59 +63,84 @@ public void init(ZeppelinConfiguration conf) throws
IOException {
oneWaySync = conf.getBoolean(ConfVars.ZEPPELIN_NOTEBOOK_ONE_WAY_SYNC);
String allStorageClassNames = conf.getNotebookStorageClass().trim();
if (allStorageClassNames.isEmpty()) {
- allStorageClassNames = defaultStorage;
+ allStorageClassNames = DEFAULT_STORAGE;
LOGGER.warn("Empty ZEPPELIN_NOTEBOOK_STORAGE conf parameter, using
default {}",
- defaultStorage);
+ DEFAULT_STORAGE);
}
String[] storageClassNames = allStorageClassNames.split(",");
if (storageClassNames.length > getMaxRepoNum()) {
LOGGER.warn("Unsupported number {} of storage classes in
ZEPPELIN_NOTEBOOK_STORAGE : {}\n" +
"first {} will be used", storageClassNames.length,
allStorageClassNames, getMaxRepoNum());
}
+// init the underlying NotebookRepo
for (int i = 0; i < Math.min(storageClassNames.length, getMaxRepoNum());
i++) {
NotebookRepo notebookRepo =
PluginManager.get().loadNotebookRepo(storageClassNames[i].trim());
if (notebookRepo != null) {
notebookRepo.init(conf);
repos.add(notebookRepo);
}
}
+
// couldn't initialize any storage, use default
if (getRepoCount() == 0) {
- LOGGER.info("No storage could be initialized, using default {} storage",
defaultStorage);
- NotebookRepo defaultNotebookRepo =
PluginManager.get().loadNotebookRepo(defaultStorage);
+ LOGGER.info("No storage could be initialized, using default {} storage",
DEFAULT_STORAGE);
+ NotebookRepo defaultNotebookRepo =
PluginManager.get().loadNotebookRepo(DEFAULT_STORAGE);
defaultNotebookRepo.init(conf);
repos.add(defaultNotebookRepo);
}
+// sync for anonymous mode on start
+if (getRepoCount() > 1 &&
conf.getBoolean(ConfVars.ZEPPELIN_ANONYMOUS_ALLOWED)) {
+ try {
+sync(AuthenticationInfo.ANONYMOUS);
+ } catch (IOException e) {
+LOGGER.error("Couldn't sync anonymous mode on start ", e);
+ }
+}
+ }
+
+ // Zeppelin change its note file name structure in 0.9.0, this is called
when upgrading
+ // from 0.9.0 before to 0.9.0 after
+ public void convertNoteFiles(ZeppelinConfiguration conf, boolean deleteOld)
throws IOException {
// convert old note file (noteId/note.json) to new note file
(note_name_note_id.zpln)
-boolean convertToNew =
conf.getBoolean(ConfVars.ZEPPELIN_NOTEBOOK_NEW_FORMAT_CONVERT);
-boolean deleteOld =
conf.getBoolean(ConfVars.ZEPPELIN_NOTEBOOK_NEW_FORMAT_DELETE_OLD);
-if (convertToNew) {
- NotebookRepo newNotebookRepo = repos.get(0);
+for (int i = 0; i < repos.size(); ++i) {
+ NotebookRepo newNotebookRepo = repos.get(i);
OldNotebookRepo oldNotebookRepo =
-
PluginManager.get().loadOldNotebookRepo(newNotebookRepo.getClass().getCanonicalName());
+
PluginManager.get().loadOldNotebookRepo(newNotebookRepo.getClass().getCanonicalName());
oldNotebookRepo.init(conf);
List oldNotesInfo =
oldNotebookRepo.list(AuthenticationInfo.ANONYMOUS);
LOGGER.info("Convert old note file to new style, note count: " +
oldNotesInfo.size());
+ LOGGER.info("Delete old note: " + deleteOld);
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
