[jira] [Created] (ZEPPELIN-4228) Interpreter name contains spaces and cannot be deleted
liuxun created ZEPPELIN-4228: Summary: Interpreter name contains spaces and cannot be deleted Key: ZEPPELIN-4228 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4228 Project: Zeppelin Issue Type: Bug Reporter: liuxun Add new interpreter type, this interpreter type cannot be deleted if the interpreter name contains spaces -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZEPPELIN-3649) When multiple people edit the same note, there is no prompt message.
liuxun created ZEPPELIN-3649: Summary: When multiple people edit the same note, there is no prompt message. Key: ZEPPELIN-3649 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3649 Project: Zeppelin Issue Type: Improvement Components: zeppelin-server Reporter: liuxun When multiple people edit the same note through different zeppelin-web, zeppelin has no prompt message. This way the notes are easily covered by different people, resulting in incomplete notes. I suggest that when the first person opens note-1, if a second person also opens note-1, the second person's note should prompt, "This note is being edited by someone", so that You can avoid two people writing the same note at the same time, covering each other, causing the loss of note content. Please discuss whether you need to implement this feature. If need to implement this feature, I am glad to. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZEPPELIN-3626) Cluster management and client module design
liuxun created ZEPPELIN-3626: Summary: Cluster management and client module design Key: ZEPPELIN-3626 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3626 Project: Zeppelin Issue Type: Sub-task Components: zeppelin-server Affects Versions: 0.9.0 Reporter: liuxun Assignee: liuxun h4. Cluster management service The cluster management service uses the Raft algorithm library copycatServer to form a service cluster with consistent service status in the Zeppelin cluster. # The cluster management service runs in each Zeppelin-Server; # The cluster management service establishes a cluster by using the copycatServer class of the Raft algorithm library, maintains the ClusterStateMachine, and manages the service state metadata of each Zeppelin-Server through the PutCommand, GetQuery, and DeleteCommand operation commands. # Launch the Thrift service in the cluster management service to enable the cluster interpreter process to be created by remote calls in each Zeppelin-Server; h4. Cluster management client The cluster management client connects to the cluster management service for metadata operations of services and processes through the Raft algorithm library copycatClient. # The cluster management client runs in each Zeppelin-Server and Zeppelin Interpreter process; # The cluster management client manages the Zeppelin-Server and Zeppelin Interpreter process state (metadata information) in the ClusterStateMachine by using the copycatClient class of the Raft library to connect to the copycatServer. When the Zeppelin-Server and Zeppelin Interpreter processes are started, They are added to the ClusterStateMachine and are removed from the ClusterStateMachine when the Zeppelin-Server and Zeppelin Interpreter processes are closed. # In a distributed environment, network anomalies, network delays, or service exceptions may occur. After copycatClient submits metadata to the cluster, it checks whether the submission is successful. After the submission fails, the metadata is saved in the local message queue. Retrying by copycatClient through a separate commit thread; h4. Cluster monitoring module The cluster monitoring module checks if each Zeppelin-Server and Zeppelin Interpreter process in the cluster is active # The cluster monitoring module runs in each Zeppelin-Server and Zeppelin Interpreter process, periodically sending heartbeat data of the service or process to the cluster; # When the cluster monitoring module runs in Zeppelin-Server, it collects the CPU and MEMORY usage of the server, and sends the resource usage rate to the cluster's ClusterStateMachine. When the cluster interpreter process needs to be created, the server is idle from the resource. Created in ; # Resource usage statistics strategy, in order to avoid the instantaneous high peak and low peak of the server, the cluster monitoring will collect the average resource usage in the most recent period for reporting, and improve the reasonable line and effectiveness of the server resources as much as possible; # When the cluster monitoring module runs in Zeppelin-Server, it checks the heartbeat data of each Zeppelin-Server and Zeppelin Interpreter process. If it times out, it considers that the service or process is abnormally unavailable and removes it from the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZEPPELIN-3623) Create a cluster interpreter process module design
liuxun created ZEPPELIN-3623:
Summary: Create a cluster interpreter process module design
Key: ZEPPELIN-3623
URL: https://issues.apache.org/jira/browse/ZEPPELIN-3623
Project: Zeppelin
Issue Type: Sub-task
Components: zeppelin-server
Affects Versions: 0.9.0
Reporter: liuxun
Assignee: liuxun
Fix For: 0.9.0
h3.
# Create an interpreter process in the specified Zeeppelin-Server via Thrift
{{zeppelin/zeppelin-interpreter/src/main/thrift/ClusterManagerService.thrift}}
# Create interpreter process parameters remotely
struct ClusterInterpreterParam {
1: string host,
2: i32 port,
3: string userName,
4: string noteId,
5: string replName
}
# Create a cluster interpreter process function
service ClusterManagerService {
bool startInterpreterProcess(1: ClusterInterpreterParam
clusterInterpreterParam);
}
The Interpreter process in Zeppelin-Server is created or used in the
getOrCreateClusterIntpProcess() function of the ManagedInterpreterGroup class,
so the process of creating the cluster interpreter process is put into this
function;
*The logic for creating a cluster interpreter is as follows*:
** The user performs a note operation on the Zepplin-Server-A server;
** If the remoteInterpreterProcess instance is not created in the
getOrCreateClusterIntpProcess() function, find the Zepplin-Server with the idle
resources from the metadata of the cluster.
** If the server with idle resources is the Zepplin-Server-A native, the
Interpreter Process instance is created directly locally;
** If the server with idle resources is not Zepplin-Server-A, Zepplin-Server-A
calls the callStartInterpreterProcess() function interface of the
Zepplin-Server-B server through the Thrift interface, and the Interpreter
Process instance is performed by getOrCreateClusterIntpProcess() in the
Zepplin-Server-B server. Creation.
** In particular, the remote creation process needs to be aware that because
the Interpreter Process instance is created locally or remotely in the same
code logic, in order to avoid the getOrCreateClusterIntpProcess() infinite loop
call, a createClusterInptProcess boolean variable is added, if there are
multiple Zepplin-Servers. The composedClusterInptProcess is equal to true in
the cluster environment. It is equal to false in the single server environment.
The createClusterInptProcess boolean variable is set to false in the
Zepplin-Server-B server when the callStartInterpreterProcess() function is
called, so getOrCreateClusterIntpProcess() It is possible to create an
Interpreter Process instance locally on the Zepplin-Server-B server;
** After the Interpreter Process instance is created locally on the
Zepplin-Server-B server, the Interpreter Process instance will report its
Thrift IP and port to the Zeppelin cluster metadata. The
remoteInterpreterProcess in the Zepplin-Server-A server is connected to the
cluster metadata. The Thrift IP and port can establish a remote connection
across servers;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ZEPPELIN-3613) Cluster Session module design
liuxun created ZEPPELIN-3613: Summary: Cluster Session module design Key: ZEPPELIN-3613 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3613 Project: Zeppelin Issue Type: Sub-task Components: zeppelin-server Affects Versions: 0.9.0 Reporter: liuxun Fix For: 0.9.0 h3. Since the Interpreter process in Zeppelin is a resident process, the Zeppelin-Server service natively saves the runtime environment of the user, Note, and the Thrift remote connection of the corresponding interpreter process through the Session, but in a distributed environment, you It is not possible to determine which Zeppelin-Server service the user will log in to through the domain name, so the Zeppelin-Server service must be able to support users who are still on the A server to continue to use it in the B server. For this reason, we have two implementations: # Cluster unified cache Session The Session created in each Zeppelin-Server service is stored in a unified manner through Cluster MetaData, and the session information is obtained from Cluster MetaData regardless of which Zeppelin-Server service the user logs in. Advantages and Disadvantages: The code size is large, and it is complicated to synchronize the session state logic in each server. The metadata stored in Cluster MetaData is also more; # Rebuild Session The user who originally uses the A server will have the user's session. When the user logs in to the B server, the Zeppelin-Server service of the B server re-establishes the user in the B server according to the user and the Note information combined with the Interpreter metadata in the Cluster MetaData. Session Advantages and Disadvantages: The code implementation is relatively simple, and there is no need to synchronize the session state in each server. The process of rebuilding the Session is very fast and simple; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZEPPELIN-3612) Cluster High availability module design
liuxun created ZEPPELIN-3612: Summary: Cluster High availability module design Key: ZEPPELIN-3612 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3612 Project: Zeppelin Issue Type: Sub-task Components: zeppelin-server Affects Versions: 0.9.0 Reporter: liuxun Fix For: 0.9.0 h3. In the case of a partial Zeppelin-Server service process or server exception, the service can continue to be served; the Zeppelin-Server service can sense the availability of all Interpreter processes in the server cluster; # *Raft protocol* The Raft protocol ensures that only N/2+1 servers in the cluster need to be in a normal state without affecting the service. # *Interpreter process monitoring* The Interpreter process creates a process heartbeat thread through the ClusterMonitor class, and periodically sends the Interpreter process heartbeat information and the IP and port information of the Thrift interface to the cluster. When the Interpreter process is closed, the process of deleting the process metadata is sent to the cluster. The process health check thread is created in the Zeppelin-Server through the ClusterMonitor class. The heartbeat of all Interpreter processes in the Cluster MetaData is periodically checked. If the timeout expires, the process metadata is deleted to prevent the Interpreter process from being abnormal. # *Interpreter process rebuild* When the interpreter process is created, the Zeppelin-Server detects the session information of the Interpreter process. First, it checks whether the process is valid. If it is not available, the corresponding session is cleared and the Interpreter process is re-created. Preventing an Interpreter process from being abnormal or a server exception on the process causes Interpreter to be unavailable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZEPPELIN-3611) High utilization module design
liuxun created ZEPPELIN-3611: Summary: High utilization module design Key: ZEPPELIN-3611 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3611 Project: Zeppelin Issue Type: Sub-task Components: zeppelin-server Affects Versions: 0.9.0 Reporter: liuxun Fix For: 0.9.0 h3. When you create an Interpreter process in Zeppelin-Server, you can create an Interpreter process in another Zeppelin-Server from a remote server with cluster resources, and establish a session. # *Server resource reporting* Zeppelin-Server collects server hardware CPU and memory usage, periodically reports to the cluster, and stores metadata in Cluster MetaData. # *Create an interpreter process remotely* When Zeppelin-Server creates an Interpreter process, first find out the server with idle resources from Cluster MetaData, and create an Interpreter process remotely through Thrift. # *Reuse the Interpreter process* When Zeppelin-Server creates an Interpreter process, it first looks up the Cluster MetaData to see if the required Interpreter process already exists. If it already exists, it obtains the IP and port of the Interpreter process's Thrift through the Cluster MetaData to create the user's note and Interpreter. Session association; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZEPPELIN-3610) Cluster Raft module design
liuxun created ZEPPELIN-3610: Summary: Cluster Raft module design Key: ZEPPELIN-3610 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3610 Project: Zeppelin Issue Type: Sub-task Components: zeppelin-server Affects Versions: 0.9.0 Reporter: liuxun Fix For: 0.9.0 By using the Raft protocol, multiple Zeppelin-Server groups are built into a Zeppelin cluster, the cluster State Machine is maintained through the Raft protocol, and the services in the cluster are agreed upon. The Zeppelin-Server and Zeppelin-Interperter services and processes are stored in the Cluster MetaData. Metadata information; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZEPPELIN-3471) Needs clustering features like hdfs to improve the availability of zeppelin
liuxun created ZEPPELIN-3471: Summary: Needs clustering features like hdfs to improve the availability of zeppelin Key: ZEPPELIN-3471 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3471 Project: Zeppelin Issue Type: New Feature Components: zeppelin-server Reporter: liuxun # At present, Zeppelin's Interpreter is only started on the same machine of the Zeppelin server. It cannot provide load balancing capability and more reasonable server resource utilization for high availability and a large number of users. # Zeppelin-server needs to provide multiple copy services on multiple servers; # Each zeppelin-server manages the life cycle of Interpreter processes running on this machine; # The Interpreter process can be started on any server through any zeppelin-web; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
