ZEPPELIN-2619. Save note in ${notename_noteid}.zpln instead of
${noteid}/note.json
### What is this PR for?
This PR has serverla main changes:
1. `NoteManager` is the most importance class that would build the filesystem
tree structure i memory and all note related operation is handled by this class.
2. `NotebookRepo` add several new methods for folder operations, like moving
folder, removing folder.
3. The semantics of `name` of `Note` is changed. It is changed to the note name
instead of the full note path, there's one transiant field `path` which
represent the path of note.
Design doc
https://docs.google.com/document/d/126egAQmhQOL4ynxJ3AQJQRBBLdW8TATYcGkDL1DNZoE/edit?usp=sharing
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://github.com/zjffdu/zeppelin/tree/ZEPPELIN-2619
### How should this be tested?
* Ci pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <[email protected]>
Closes #3163 from zjffdu/ZEPPELIN-2619 and squashes the following commits:
ebcf4f87b [Jeff Zhang] ZEPPELIN-2619. Save note in [Title].zpln instead of
[NOTEID]/note.json
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/085efeb6
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/085efeb6
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/085efeb6
Branch: refs/heads/master
Commit: 085efeb646f3a95b39bf555b5b04551119e3c0b9
Parents: dabb7d9
Author: Jeff Zhang <[email protected]>
Authored: Fri Jul 6 15:01:44 2018 +0800
Committer: Jeff Zhang <[email protected]>
Committed: Thu Sep 20 10:34:38 2018 +0800
----------------------------------------------------------------------
notebook/2A94M5J1Z/note.json | 376 -----
notebook/2BWJFTXKJ/note.json | 1309 ------------------
notebook/2BYEZ5EVK/note.json | 887 ------------
notebook/2C2AUG798/note.json | 779 -----------
notebook/2C35YU814/note.json | 806 -----------
notebook/2C57UKYWR/note.json | 334 -----
.../Basic Features (Spark)_2A94M5J1Z.zpln | 376 +++++
...Python \342\200\242 PySpark)_2C2AUG798.zpln" | 779 +++++++++++
.../Zeppelin Tutorial/R (SparkR)_2BWJFTXKJ.zpln | 1309 ++++++++++++++++++
...ng Flink for batch processing_2C35YU814.zpln | 806 +++++++++++
.../Using Mahout_2BYEZ5EVK.zpln | 887 ++++++++++++
.../Using Pig for querying data_2C57UKYWR.zpln | 334 +++++
.../zeppelin/scheduler/SchedulerFactory.java | 2 +-
.../notebook/repo/AzureNotebookRepo.java | 135 +-
.../notebook/repo/FileSystemNotebookRepo.java | 94 +-
.../repo/FileSystemNotebookRepoTest.java | 54 +-
.../src/test/resources/log4j.properties | 22 +
.../zeppelin/notebook/repo/GCSNotebookRepo.java | 62 +-
.../notebook/repo/GCSNotebookRepoTest.java | 53 +-
.../zeppelin/notebook/repo/GitNotebookRepo.java | 101 +-
.../notebook/repo/GitNotebookRepoTest.java | 136 +-
.../notebook/repo/NotebookRepoSyncTest.java | 174 +--
.../git/src/test/resources/2A94M5J1Z/note.json | 369 -----
.../git/src/test/resources/2A94M5J2Z/note.json | 95 --
.../notebook/my_project/my_note1_2A94M5J1Z.zpln | 369 +++++
.../notebook/my_project/my_note2_2A94M5J2Z.zpln | 95 ++
.../notebook/repo/GitHubNotebookRepo.java | 7 +-
.../notebook/repo/GitHubNotebookRepoTest.java | 27 +-
.../src/test/resources/2A94M5J1Z/note.json | 369 -----
.../src/test/resources/2A94M5J2Z/note.json | 95 --
.../notebook/my_project/my_note1_2A94M5J1Z.zpln | 369 +++++
zeppelin-plugins/notebookrepo/mongodb/pom.xml | 63 -
.../notebook/repo/MongoNotebookRepo.java | 222 ---
.../zeppelin/notebook/repo/S3NotebookRepo.java | 83 +-
.../zeppelin/notebook/repo/VFSNotebookRepo.java | 263 ++--
.../notebook/repo/TestVFSNotebookRepo.java | 113 --
.../notebook/repo/VFSNotebookRepoTest.java | 129 ++
.../repo/zeppelinhub/ZeppelinHubRepo.java | 40 +-
zeppelin-plugins/pom.xml | 1 -
.../apache/zeppelin/rest/NotebookRestApi.java | 17 +-
.../apache/zeppelin/server/ZeppelinServer.java | 3 +-
.../zeppelin/service/NotebookService.java | 514 +++++--
.../zeppelin/service/SimpleServiceCallback.java | 6 +-
.../zeppelin/socket/ConnectionManager.java | 3 +-
.../apache/zeppelin/socket/NotebookServer.java | 836 ++++-------
.../apache/zeppelin/recovery/RecoveryTest.java | 18 +-
.../zeppelin/rest/AbstractTestRestApi.java | 18 +-
.../zeppelin/rest/InterpreterRestApiTest.java | 5 +-
.../zeppelin/rest/NotebookRestApiTest.java | 32 +-
.../zeppelin/rest/ZeppelinRestApiTest.java | 62 +-
.../zeppelin/rest/ZeppelinSparkClusterTest.java | 62 +-
.../zeppelin/service/NotebookServiceTest.java | 244 +++-
.../zeppelin/socket/NotebookServerTest.java | 106 +-
.../src/test/resources/log4j.properties | 1 +
zeppelin-web/src/app/home/home.controller.js | 2 +-
.../src/app/notebook/notebook.controller.js | 6 +-
.../array-ordering/array-ordering.service.js | 4 +-
.../note-action/note-action.service.js | 14 +-
.../note-create/note-create.controller.js | 8 +-
.../components/note-list/note-list.factory.js | 8 +-
.../websocket/websocket-message.service.js | 20 +-
zeppelin-zengine/pom.xml | 16 -
.../helium/HeliumApplicationFactory.java | 19 +-
.../interpreter/InterpreterSettingManager.java | 86 +-
.../zeppelin/notebook/FileSystemStorage.java | 35 +
.../org/apache/zeppelin/notebook/Folder.java | 257 ----
.../zeppelin/notebook/FolderListener.java | 24 -
.../apache/zeppelin/notebook/FolderView.java | 239 ----
.../java/org/apache/zeppelin/notebook/Note.java | 353 ++---
.../notebook/NoteEventAsyncListener.java | 228 +++
.../zeppelin/notebook/NoteEventListener.java | 16 +-
.../org/apache/zeppelin/notebook/NoteInfo.java | 38 +-
.../apache/zeppelin/notebook/NoteManager.java | 569 ++++++++
.../zeppelin/notebook/NoteNameListener.java | 30 -
.../org/apache/zeppelin/notebook/Notebook.java | 613 ++++----
.../notebook/NotebookAuthorization.java | 46 +-
.../notebook/NotebookEventListener.java | 27 -
.../org/apache/zeppelin/notebook/Paragraph.java | 21 +-
.../notebook/repo/InMemoryNotebookRepo.java | 97 ++
.../zeppelin/notebook/repo/NotebookRepo.java | 130 +-
.../notebook/repo/NotebookRepoSync.java | 195 +--
.../repo/NotebookRepoWithVersionControl.java | 30 +-
.../zeppelin/scheduler/RemoteScheduler.java | 2 -
.../apache/zeppelin/search/LuceneSearch.java | 23 +-
.../apache/zeppelin/search/SearchService.java | 67 +-
.../helium/HeliumApplicationFactoryTest.java | 12 +-
.../interpreter/AbstractInterpreterTest.java | 2 +-
.../apache/zeppelin/notebook/FolderTest.java | 214 ---
.../zeppelin/notebook/FolderViewTest.java | 358 -----
.../zeppelin/notebook/NoteManagerTest.java | 66 +
.../org/apache/zeppelin/notebook/NoteTest.java | 87 +-
.../apache/zeppelin/notebook/NotebookTest.java | 357 ++---
.../apache/zeppelin/notebook/ParagraphTest.java | 22 +-
.../zeppelin/search/LuceneSearchTest.java | 96 +-
94 files changed, 9249 insertions(+), 9639 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/085efeb6/notebook/2A94M5J1Z/note.json
----------------------------------------------------------------------
diff --git a/notebook/2A94M5J1Z/note.json b/notebook/2A94M5J1Z/note.json
deleted file mode 100644
index 6e8e06f..0000000
--- a/notebook/2A94M5J1Z/note.json
+++ /dev/null
@@ -1,376 +0,0 @@
-{
- "paragraphs": [
- {
- "text": "%md\n## Welcome to Zeppelin.\n##### This is a live tutorial,
you can run the code yourself. (Shift-Enter to Run)",
- "user": "anonymous",
- "dateUpdated": "Dec 17, 2016 3:32:15 PM",
- "config": {
- "colWidth": 12.0,
- "editorHide": true,
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false,
- "keys": [],
- "values": [],
- "groups": [],
- "scatter": {}
- }
- }
- ],
- "enabled": true,
- "editorSetting": {
- "language": "markdown",
- "editOnDblClick": true
- },
- "editorMode": "ace/mode/markdown",
- "tableHide": false
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cdiv
class\u003d\"markdown-body\"\u003e\n\u003ch2\u003eWelcome to
Zeppelin.\u003c/h2\u003e\n\u003ch5\u003eThis is a live tutorial, you can run
the code yourself. (Shift-Enter to Run)\u003c/h5\u003e\n\u003c/div\u003e"
- }
- ]
- },
- "apps": [],
- "jobName": "paragraph_1423836981412_-1007008116",
- "id": "20150213-231621_168813393",
- "dateCreated": "Feb 13, 2015 11:16:21 PM",
- "dateStarted": "Dec 17, 2016 3:32:15 PM",
- "dateFinished": "Dec 17, 2016 3:32:18 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "title": "Load data into table",
- "text": "import org.apache.commons.io.IOUtils\nimport
java.net.URL\nimport java.nio.charset.Charset\n\n// Zeppelin creates and
injects sc (SparkContext) and sqlContext (HiveContext or SqlContext)\n// So you
don\u0027t need create them manually\n\n// load bank data\nval bankText \u003d
sc.parallelize(\n IOUtils.toString(\n new
URL(\"https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv\";),\n
Charset.forName(\"utf8\")).split(\"\\n\"))\n\ncase class Bank(age: Integer,
job: String, marital: String, education: String, balance: Integer)\n\nval bank
\u003d bankText.map(s \u003d\u003e s.split(\";\")).filter(s \u003d\u003e s(0)
!\u003d \"\\\"age\\\"\").map(\n s \u003d\u003e Bank(s(0).toInt, \n
s(1).replaceAll(\"\\\"\", \"\"),\n s(2).replaceAll(\"\\\"\",
\"\"),\n s(3).replaceAll(\"\\\"\", \"\"),\n
s(5).replaceAll(\"\\\"\", \"\").toInt\n
)\n).toDF()\nbank.registerTempTable(\"bank\")",
- "user": "anonymous",
- "dateUpdated": "Dec 17, 2016 3:30:09 PM",
- "config": {
- "colWidth": 12.0,
- "title": true,
- "enabled": true,
- "editorMode": "ace/mode/scala",
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false
- }
- }
- ],
- "editorSetting": {
- "language": "scala",
- "editOnDblClick": false
- }
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "import org.apache.commons.io.IOUtils\nimport
java.net.URL\nimport java.nio.charset.Charset\nbankText:
org.apache.spark.rdd.RDD[String] \u003d ParallelCollectionRDD[36] at
parallelize at \u003cconsole\u003e:43\ndefined class Bank\nbank:
org.apache.spark.sql.DataFrame \u003d [age: int, job: string ... 3 more
fields]\nwarning: there were 1 deprecation warning(s); re-run with -deprecation
for details\n"
- }
- ]
- },
- "apps": [],
- "jobName": "paragraph_1423500779206_-1502780787",
- "id": "20150210-015259_1403135953",
- "dateCreated": "Feb 10, 2015 1:52:59 AM",
- "dateStarted": "Dec 17, 2016 3:30:09 PM",
- "dateFinished": "Dec 17, 2016 3:30:58 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%sql \nselect age, count(1) value\nfrom bank \nwhere age \u003c
30 \ngroup by age \norder by age",
- "user": "anonymous",
- "dateUpdated": "Mar 17, 2017 12:18:02 PM",
- "config": {
- "colWidth": 4.0,
- "results": [
- {
- "graph": {
- "mode": "multiBarChart",
- "height": 366.0,
- "optionOpen": false
- },
- "helium": {}
- }
- ],
- "enabled": true,
- "editorSetting": {
- "language": "sql",
- "editOnDblClick": false
- },
- "editorMode": "ace/mode/sql"
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data":
"age\tvalue\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n"
- }
- ]
- },
- "apps": [],
- "jobName": "paragraph_1423500782552_-1439281894",
- "id": "20150210-015302_1492795503",
- "dateCreated": "Feb 10, 2015 1:53:02 AM",
- "dateStarted": "Dec 17, 2016 3:30:13 PM",
- "dateFinished": "Dec 17, 2016 3:31:04 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%sql \nselect age, count(1) value \nfrom bank \nwhere age
\u003c ${maxAge\u003d30} \ngroup by age \norder by age",
- "user": "anonymous",
- "dateUpdated": "Mar 17, 2017 12:17:39 PM",
- "config": {
- "colWidth": 4.0,
- "results": [
- {
- "graph": {
- "mode": "multiBarChart",
- "height": 294.0,
- "optionOpen": false
- },
- "helium": {}
- }
- ],
- "enabled": true,
- "editorSetting": {
- "language": "sql",
- "editOnDblClick": false
- },
- "editorMode": "ace/mode/sql"
- },
- "settings": {
- "params": {
- "maxAge": "35"
- },
- "forms": {
- "maxAge": {
- "name": "maxAge",
- "defaultValue": "30",
- "hidden": false
- }
- }
- },
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data":
"age\tvalue\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n30\t150\n31\t199\n32\t224\n33\t186\n34\t231\n"
- }
- ]
- },
- "apps": [],
- "jobName": "paragraph_1423720444030_-1424110477",
- "id": "20150212-145404_867439529",
- "dateCreated": "Feb 12, 2015 2:54:04 PM",
- "dateStarted": "Dec 17, 2016 3:30:58 PM",
- "dateFinished": "Dec 17, 2016 3:31:07 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%sql \nselect age, count(1) value \nfrom bank \nwhere
marital\u003d\"${marital\u003dsingle,single|divorced|married}\" \ngroup by age
\norder by age",
- "user": "anonymous",
- "dateUpdated": "Mar 17, 2017 12:18:18 PM",
- "config": {
- "colWidth": 4.0,
- "results": [
- {
- "graph": {
- "mode": "stackedAreaChart",
- "height": 280.0,
- "optionOpen": false
- },
- "helium": {}
- }
- ],
- "enabled": true,
- "editorSetting": {
- "language": "sql",
- "editOnDblClick": false
- },
- "editorMode": "ace/mode/sql"
- },
- "settings": {
- "params": {
- "marital": "single"
- },
- "forms": {
- "marital": {
- "name": "marital",
- "defaultValue": "single",
- "options": [
- {
- "value": "single"
- },
- {
- "value": "divorced"
- },
- {
- "value": "married"
- }
- ],
- "hidden": false
- }
- }
- },
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data":
"age\tvalue\n19\t4\n20\t3\n21\t7\n22\t9\n23\t17\n24\t13\n25\t33\n26\t56\n27\t64\n28\t78\n29\t56\n30\t92\n31\t86\n32\t105\n33\t61\n34\t75\n35\t46\n36\t50\n37\t43\n38\t44\n39\t30\n40\t25\n41\t19\n42\t23\n43\t21\n44\t20\n45\t15\n46\t14\n47\t12\n48\t12\n49\t11\n50\t8\n51\t6\n52\t9\n53\t4\n55\t3\n56\t3\n57\t2\n58\t7\n59\t2\n60\t5\n66\t2\n69\t1\n"
- }
- ]
- },
- "apps": [],
- "jobName": "paragraph_1423836262027_-210588283",
- "id": "20150213-230422_1600658137",
- "dateCreated": "Feb 13, 2015 11:04:22 PM",
- "dateStarted": "Dec 17, 2016 3:31:05 PM",
- "dateFinished": "Dec 17, 2016 3:31:09 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%md\n## Congratulations, it\u0027s done.\n##### You can create
your own notebook in \u0027Notebook\u0027 menu. Good luck!",
- "user": "anonymous",
- "dateUpdated": "Dec 17, 2016 3:30:24 PM",
- "config": {
- "colWidth": 12.0,
- "editorHide": true,
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false
- }
- }
- ],
- "enabled": true,
- "editorSetting": {
- "language": "markdown",
- "editOnDblClick": true
- },
- "editorMode": "ace/mode/markdown",
- "tableHide": false
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cdiv
class\u003d\"markdown-body\"\u003e\n\u003ch2\u003eCongratulations,
it\u0026rsquo;s done.\u003c/h2\u003e\n\u003ch5\u003eYou can create your own
notebook in \u0026lsquo;Notebook\u0026rsquo; menu. Good
luck!\u003c/h5\u003e\n\u003c/div\u003e"
- }
- ]
- },
- "apps": [],
- "jobName": "paragraph_1423836268492_216498320",
- "id": "20150213-230428_1231780373",
- "dateCreated": "Feb 13, 2015 11:04:28 PM",
- "dateStarted": "Dec 17, 2016 3:30:24 PM",
- "dateFinished": "Dec 17, 2016 3:30:29 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "text": "%md\n\nAbout bank data\n\n```\nCitation Request:\n This
dataset is public available for research. The details are described in [Moro et
al., 2011]. \n Please include this citation if you plan to use this
database:\n\n [Moro et al., 2011] S. Moro, R. Laureano and P. Cortez. Using
Data Mining for Bank Direct Marketing: An Application of the CRISP-DM
Methodology. \n In P. Novais et al. (Eds.), Proceedings of the European
Simulation and Modelling Conference - ESM\u00272011, pp. 117-121, Guimarães,
Portugal, October, 2011. EUROSIS.\n\n Available at: [pdf]
http://hdl.handle.net/1822/14838\n [bib]
http://www3.dsi.uminho.pt/pcortez/bib/2011-esm-1.txt\n```";,
- "user": "anonymous",
- "dateUpdated": "Dec 17, 2016 3:30:34 PM",
- "config": {
- "colWidth": 12.0,
- "editorHide": true,
- "results": [
- {
- "graph": {
- "mode": "table",
- "height": 300.0,
- "optionOpen": false
- }
- }
- ],
- "enabled": true,
- "editorSetting": {
- "language": "markdown",
- "editOnDblClick": true
- },
- "editorMode": "ace/mode/markdown",
- "tableHide": false
- },
- "settings": {
- "params": {},
- "forms": {}
- },
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cdiv
class\u003d\"markdown-body\"\u003e\n\u003cp\u003eAbout bank
data\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003eCitation Request:\n This
dataset is public available for research. The details are described in [Moro et
al., 2011]. \n Please include this citation if you plan to use this
database:\n\n [Moro et al., 2011] S. Moro, R. Laureano and P. Cortez. Using
Data Mining for Bank Direct Marketing: An Application of the CRISP-DM
Methodology. \n In P. Novais et al. (Eds.), Proceedings of the European
Simulation and Modelling Conference - ESM\u0026#39;2011, pp. 117-121,
Guimarães, Portugal, October, 2011. EUROSIS.\n\n Available at: [pdf]
http://hdl.handle.net/1822/14838\n [bib]
http://www3.dsi.uminho.pt/pcortez/bib/2011-esm-1.txt\n\u003c/code\u003e\u003c/pre\u003e\n\u003c/div\u003e";
- }
- ]
- },
- "apps": [],
- "jobName": "paragraph_1427420818407_872443482",
- "id": "20150326-214658_12335843",
- "dateCreated": "Mar 26, 2015 9:46:58 PM",
- "dateStarted": "Dec 17, 2016 3:30:34 PM",
- "dateFinished": "Dec 17, 2016 3:30:34 PM",
- "status": "FINISHED",
- "progressUpdateIntervalMs": 500
- },
- {
- "config": {},
- "settings": {
- "params": {},
- "forms": {}
- },
- "apps": [],
- "jobName": "paragraph_1435955447812_-158639899",
- "id": "20150703-133047_853701097",
- "dateCreated": "Jul 3, 2015 1:30:47 PM",
- "status": "READY",
- "progressUpdateIntervalMs": 500
- }
- ],
- "name": "Zeppelin Tutorial/Basic Features (Spark)",
- "id": "2A94M5J1Z",
- "angularObjects": {
- "2C73DY9P9:shared_process": []
- },
- "config": {
- "looknfeel": "default"
- },
- "info": {}
-}
\ No newline at end of file
