We want to have a Zeppelin service that serves over 200 people in our company. So we plan to have around 10 – 15 Zeppelin instances behind an ELB. We use S3 as notebook storage, and hence all our Zeppelin instances are referring to the same S3 location for notebooks. But there is one thing that breaks the whole thing: Zeppelin is storing the notebook authorization information into a LOCAL file called notebook-authorization.json. In order to solve the problem we setup some NFS like thing to let every Zeppelin instance to refer to the same configuration location through FS mount. The method has following problems:
1. We cannot handle concurrency conditions where multiple Zeppelin instances are editing the files at the same time. Some unexpected behaviors will happen. 2. I found out that Zeppelin only reads the notebook-authorization.json file to memory on startup. After startup, it only treats the authorization in memory as the source of truth. Zeppelin will never read that file anymore unless you restart it. It only writes to it, from memory. Therefore even without the concurrency problem described in (1), it is not able to get the correct authorization for notebooks after other Zeppelin instances change the authorization file. I know the reasons behind for making authorizations separate from notebook but it actually brings up more serious problems like this. Any ideas how to tackle this problem and make Zeppelin scalable?
