[
https://issues.apache.org/jira/browse/KYLIN-4446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17097149#comment-17097149
]
ASF GitHub Bot commented on KYLIN-4446:
---------------------------------------
sanjulian commented on a change in pull request #1182:
URL: https://github.com/apache/kylin/pull/1182#discussion_r417748474
##########
File path: kubernetes/docker/hadoop-client/README.md
##########
@@ -0,0 +1,18 @@
+## Background
+### What is hadoop-client docker image and why we need this?
+As we all know, the node you want to deploy Kylin, should has provided Hadoop
+dependency(jars and configuration files), these dependency let you have access
+ to Hadoop Service, such as HDFS, HBase, Hive, which are needed by Apache
Kylin.
+Unfortunately, each Hadoop distribution(CHD or HDP etc.) has its own specific
jars. So, we
+can provided specific image for specific Hadoop distribution, which will make
image management task
+more easier. This will have following two benefits:
+1. Someone who has better knowledge on Hadoop can do this work, and let kylin
+ user build their Kylin image base on provided Hadoop-Client image.
+2. Upgrade Kylin will be much easier.
+
+### Build Step for CDH5.7
+1. Place Spark binary(*spark-2.3.2-bin-hadoop2.7.tgz*) into dir
`provided-binary`.
+2. Run `build-image.sh` to build image.
+
+> If you are using other hadoop distribution, please consider build image
yourself.
+
Review comment:
Could we add the some build step for apache hadoop client?
Such as download hadoop client first, and update version in dockerfile.
##########
File path: kubernetes/example/deployment/kylin-job/kylin-job-service.yaml
##########
@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: v1
+kind: Service
+metadata:
+ name: kylin-job-svc
Review comment:
I suppose we do not need the services for job, since the job pod only
for building cube instead of query.
##########
File path:
kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+ name: kylin-query
+ namespace: kylin-example
+ labels:
+ app: kylin
+ role: query
+ environment: poc
+ version: 3.0.1
+ hadoop.version: cdh5.7
+spec:
+ serviceName: kylin-query-svc
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kylin
+ role: query
+ template:
+ metadata:
+ labels:
+ app: kylin
+ role: query
+ spec:
+ hostAliases:
+ - ip: "10.1.3.90"
+ hostnames:
+ - "cdh-master"
+ - ip: "10.1.3.93"
+ hostnames:
+ - "cdh-worker-1"
+ - ip: "10.1.3.94"
+ hostnames:
+ - "cdh-worker-2"
+ - ip: "10.1.3.91"
+ hostnames:
+ - "cdh-client"
+ containers:
+ - name: kylin
+ image: kylin-client:3.0.1-cdh57
+ imagePullPolicy: IfNotPresent
+ command:
+ - sh
+ - -c
+ args:
+ - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;
Review comment:
Remove the copy before the init.
Please use subPath to mount configMap or secret files for tomcat and kylin
configuration.
Here is the reference:
https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath
##########
File path: kubernetes/docker/hadoop-client/apache-hadoop2.7/Dockerfile
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM centos:7.3.1611
+
+MAINTAINER Apache Kylin
+
+WORKDIR /tmp
+
+# install jdk and other commands
+RUN set -x \
+ && yum install -y which \
+ java-1.8.0-openjdk \
+ java-1.8.0-openjdk-devel \
+ krb5-workstation \
+ && yum clean all
+
+
+# version variables
+ARG HADOOP_VERSION=2.7.4
+ARG HIVE_VERSION=1.2.2
+ARG HBASE_VERSION=1.3.1
+ARG SPARK_VERSION=2.2.1
+ARG ZK_VERSION=3.4.6
+
+ARG APACHE_HOME=/apache
+
+RUN set -x \
+ && mkdir -p $APACHE_HOME
+
+ENV JAVA_HOME /etc/alternatives/jre
+
+# install hadoop
+ENV HADOOP_HOME=$APACHE_HOME/hadoop
+ADD hdp-clients/hadoop-${HADOOP_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/hadoop-${HADOOP_VERSION} $HADOOP_HOME
+
+#install hive
+ENV HIVE_HOME=$APACHE_HOME/hive
+ADD hdp-clients/apache-hive-${HIVE_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/apache-hive-${HIVE_VERSION} $HIVE_HOME
+
+#install hbase
+ENV HBASE_HOME=$APACHE_HOME/hbase
+ADD hdp-clients/hbase-${HBASE_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/hbase-${HBASE_VERSION} $HBASE_HOME
+
+#install spark
+ENV SPARK_HOME=$APACHE_HOME/spark
+ADD hdp-clients/spark-${SPARK_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/spark-${SPARK_VERSION} $SPARK_HOME
+
+#install zk
+ENV ZK_HOME=$APACHE_HOME/zookeeper
+ADD hdp-clients/zookeeper-${ZK_VERSION}.tar.gz $APACHE_HOME
+RUN set -x \
+ && ln -s $APACHE_HOME/zookeeper-${ZK_VERSION} $ZK_HOME
+
+ENV
PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$HBASE_HOME/bin:$ZK_HOME/bin
+
+COPY conf/hadoop $HADOOP_HOME/etc/hadoop
Review comment:
I suppose we should remove the hadoop config. There are hadoop config
for configmap in development files.
##########
File path:
kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+ name: kylin-query
+ namespace: kylin-example
+ labels:
+ app: kylin
+ role: query
+ environment: poc
+ version: 3.0.1
+ hadoop.version: cdh5.7
+spec:
+ serviceName: kylin-query-svc
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kylin
+ role: query
+ template:
+ metadata:
+ labels:
+ app: kylin
+ role: query
+ spec:
+ hostAliases:
+ - ip: "10.1.3.90"
+ hostnames:
+ - "cdh-master"
+ - ip: "10.1.3.93"
+ hostnames:
+ - "cdh-worker-1"
+ - ip: "10.1.3.94"
+ hostnames:
+ - "cdh-worker-2"
+ - ip: "10.1.3.91"
+ hostnames:
+ - "cdh-client"
+ containers:
+ - name: kylin
+ image: kylin-client:3.0.1-cdh57
+ imagePullPolicy: IfNotPresent
+ command:
+ - sh
+ - -c
+ args:
+ - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;
+ cp $KYLIN_HOME/kylin-more-config
$KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes/;
+ $TOOL_HOME/bootstrap.sh server -d;
+ ports:
+ - containerPort: 7070
+ - containerPort: 7443
+ volumeMounts:
+ - name: kylin-query-config
+ mountPath: /home/apache_kylin/kylin/conf
+ - name: tomcat-config
+ mountPath: /home/apache_kylin/kylin/tomcat-config
+ - name: kylin-more-config
+ mountPath: /home/apache_kylin/kylin/kylin-more-config/
+ - name: hadoop-config
+ mountPath: /etc/hadoop/conf
+ - name: hive-config
+ mountPath: /etc/hive/conf
+ - name: hbase-config
+ mountPath: /etc/hbase/conf
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ resources:
+ requests:
+ memory: 2Gi
+ cpu: 700m
+ limits:
+ memory: 2Gi
+ cpu: 700m
+ - name: filebeat
+ image: docker.elastic.co/beats/filebeat:6.4.3
+ args:
+ - -c
+ - /usr/share/filebeat/config/filebeat.yml
+ - -e
+ volumeMounts:
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ - name: filebeat-config
+ mountPath: /usr/share/filebeat/config
+ readOnly: true
+ resources:
+ requests:
+ memory: 1Gi
+ cpu: 300m
+ limits:
+ memory: 1Gi
+ cpu: 300m
+ volumes:
+ - name: kylin-logs
+ emptyDir:
+ sizeLimit: 3Gi
+ - name: tomcat-logs
+ emptyDir:
+ sizeLimit: 2Gi
+ - configMap:
+ name: hadoop-config
+ name: hadoop-config
+ - configMap:
+ name: hive-config
+ name: hive-config
+ - configMap:
+ name: hbase-config
+ name: hbase-config
+ - configMap:
Review comment:
As I know, there are some secret files in kylin.properties, such as
kylin.job.notification-mail-password,
kylin.security.ldap.connection-password... Could we change the kylin-***-config
as secrets instead of configMap?
##########
File path:
kubernetes/example/deployment/kylin-query/kylin-query-statefulset.yaml
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+ name: kylin-query
+ namespace: kylin-example
+ labels:
+ app: kylin
+ role: query
+ environment: poc
+ version: 3.0.1
+ hadoop.version: cdh5.7
+spec:
+ serviceName: kylin-query-svc
+ replicas: 1
+ selector:
+ matchLabels:
+ app: kylin
+ role: query
+ template:
+ metadata:
+ labels:
+ app: kylin
+ role: query
+ spec:
+ hostAliases:
+ - ip: "10.1.3.90"
+ hostnames:
+ - "cdh-master"
+ - ip: "10.1.3.93"
+ hostnames:
+ - "cdh-worker-1"
+ - ip: "10.1.3.94"
+ hostnames:
+ - "cdh-worker-2"
+ - ip: "10.1.3.91"
+ hostnames:
+ - "cdh-client"
+ containers:
+ - name: kylin
+ image: kylin-client:3.0.1-cdh57
+ imagePullPolicy: IfNotPresent
+ command:
+ - sh
+ - -c
+ args:
+ - cp $KYLIN_HOME/tomcat-config/* $KYLIN_HOME/tomcat/conf;
+ cp $KYLIN_HOME/kylin-more-config
$KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes/;
+ $TOOL_HOME/bootstrap.sh server -d;
+ ports:
+ - containerPort: 7070
+ - containerPort: 7443
+ volumeMounts:
+ - name: kylin-query-config
+ mountPath: /home/apache_kylin/kylin/conf
+ - name: tomcat-config
+ mountPath: /home/apache_kylin/kylin/tomcat-config
+ - name: kylin-more-config
+ mountPath: /home/apache_kylin/kylin/kylin-more-config/
+ - name: hadoop-config
+ mountPath: /etc/hadoop/conf
+ - name: hive-config
+ mountPath: /etc/hive/conf
+ - name: hbase-config
+ mountPath: /etc/hbase/conf
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ resources:
+ requests:
+ memory: 2Gi
+ cpu: 700m
+ limits:
+ memory: 2Gi
+ cpu: 700m
+ - name: filebeat
+ image: docker.elastic.co/beats/filebeat:6.4.3
+ args:
+ - -c
+ - /usr/share/filebeat/config/filebeat.yml
+ - -e
+ volumeMounts:
+ - name: kylin-logs
+ mountPath: /home/apache_kylin/kylin/logs
+ - name: tomcat-logs
+ mountPath: /home/apache_kylin/kylin/tomcat/logs
+ - name: filebeat-config
+ mountPath: /usr/share/filebeat/config
+ readOnly: true
+ resources:
+ requests:
+ memory: 1Gi
+ cpu: 300m
+ limits:
+ memory: 1Gi
+ cpu: 300m
+ volumes:
+ - name: kylin-logs
+ emptyDir:
+ sizeLimit: 3Gi
+ - name: tomcat-logs
+ emptyDir:
+ sizeLimit: 2Gi
+ - configMap:
Review comment:
The hadoop configmap is not used. Should we delete them or add the
volume mount for hdp configuration?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Kylin on kubernetes in a quickstart env
> ---------------------------------------
>
> Key: KYLIN-4446
> URL: https://issues.apache.org/jira/browse/KYLIN-4446
> Project: Kylin
> Issue Type: Sub-task
> Components: Integration
> Reporter: Xiaoxiang Yu
> Assignee: Xiaoxiang Yu
> Priority: Major
> Labels: kubernetes
> Fix For: v3.1.0
>
> Original Estimate: 48h
> Time Spent: 2h
> Remaining Estimate: 46h
>
> For a quick-start/PoC purpose, please consider use this template, it will
> only deploy one Kylin instance with role {color:#de350b}*ALL* {color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
