[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620:
URL: https://github.com/apache/bigtop/pull/620#discussion_r447548777
##
File path: bigtop-tests/smoke-tests/logstash/TestLogstashSmoke.groovy
##
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.junit.BeforeClass
+import org.junit.AfterClass
+import org.apache.bigtop.itest.shell.Shell
+import static org.junit.Assert.assertTrue
+import static org.junit.Assert.assertFalse
+import org.junit.Test
+import org.apache.bigtop.itest.TestUtils
+import org.junit.runner.RunWith
+import static org.junit.Assert.assertEquals
+
+class TestLogstashSmoke {
+ static Shell sh = new Shell("/bin/bash -s");
+
+ static final String LOGSTASH_CMD = "/usr/lib/logstash/bin/logstash "
+ static final String TINYBENCH_CONFIG = "resources/generator.conf "
+ static final String FILEPIPELINE_CONFIG = "resources/filepipeline.conf "
+
+ @AfterClass
+ public static void LogstashCleanUp() {
+sh.exec("rm -f resources/output.log");
+assertTrue("Logstash cleanup failed. ", sh.getRet() == 0);
+ }
+
+ @Test
+ public void LogstashTinyBenchTest() {
+/* Tiny benchmark with logstash Generator-plugin */
+sh.exec("time "
+ + LOGSTASH_CMD
+ + "-f "
+ + TINYBENCH_CONFIG
+);
+def out = sh.out.join('\n');
+assertFalse("Logstash tiny benchmark failed", (out =~ /ERROR/).find());
+ }
+
+ @Test
+ public void LogstashFilePipelineTest() {
Review comment:
Sorry for the late response.
The root cause is:
Childprocess-4.0.0 is newly released in **June 18, 2020**.
But it is not compatible with 5.4.1.
Specify childprocess to 3.0.0.
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620:
URL: https://github.com/apache/bigtop/pull/620#discussion_r442681378
##
File path: bigtop-tests/smoke-tests/logstash/TestLogstashSmoke.groovy
##
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.junit.BeforeClass
+import org.junit.AfterClass
+import org.apache.bigtop.itest.shell.Shell
+import static org.junit.Assert.assertTrue
+import static org.junit.Assert.assertFalse
+import org.junit.Test
+import org.apache.bigtop.itest.TestUtils
+import org.junit.runner.RunWith
+import static org.junit.Assert.assertEquals
+
+class TestLogstashSmoke {
+ static Shell sh = new Shell("/bin/bash -s");
+
+ static final String LOGSTASH_CMD = "/usr/lib/logstash/bin/logstash "
+ static final String TINYBENCH_CONFIG = "resources/generator.conf "
+ static final String FILEPIPELINE_CONFIG = "resources/filepipeline.conf "
+
+ @AfterClass
+ public static void LogstashCleanUp() {
+sh.exec("rm -f resources/output.log");
+assertTrue("Logstash cleanup failed. ", sh.getRet() == 0);
+ }
+
+ @Test
+ public void LogstashTinyBenchTest() {
+/* Tiny benchmark with logstash Generator-plugin */
+sh.exec("time "
+ + LOGSTASH_CMD
+ + "-f "
+ + TINYBENCH_CONFIG
+);
+def out = sh.out.join('\n');
+assertFalse("Logstash tiny benchmark failed", (out =~ /ERROR/).find());
+ }
+
+ @Test
+ public void LogstashFilePipelineTest() {
Review comment:
The PR was updated for removing LogstashFilePipelineTest and using
`$LOGSTASH_HOME `instead of a hard-coded path.
For ubuntu-16.04 build issue, let me look into it on my environment and will
give you feedback later, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620:
URL: https://github.com/apache/bigtop/pull/620#discussion_r442681378
##
File path: bigtop-tests/smoke-tests/logstash/TestLogstashSmoke.groovy
##
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.junit.BeforeClass
+import org.junit.AfterClass
+import org.apache.bigtop.itest.shell.Shell
+import static org.junit.Assert.assertTrue
+import static org.junit.Assert.assertFalse
+import org.junit.Test
+import org.apache.bigtop.itest.TestUtils
+import org.junit.runner.RunWith
+import static org.junit.Assert.assertEquals
+
+class TestLogstashSmoke {
+ static Shell sh = new Shell("/bin/bash -s");
+
+ static final String LOGSTASH_CMD = "/usr/lib/logstash/bin/logstash "
+ static final String TINYBENCH_CONFIG = "resources/generator.conf "
+ static final String FILEPIPELINE_CONFIG = "resources/filepipeline.conf "
+
+ @AfterClass
+ public static void LogstashCleanUp() {
+sh.exec("rm -f resources/output.log");
+assertTrue("Logstash cleanup failed. ", sh.getRet() == 0);
+ }
+
+ @Test
+ public void LogstashTinyBenchTest() {
+/* Tiny benchmark with logstash Generator-plugin */
+sh.exec("time "
+ + LOGSTASH_CMD
+ + "-f "
+ + TINYBENCH_CONFIG
+);
+def out = sh.out.join('\n');
+assertFalse("Logstash tiny benchmark failed", (out =~ /ERROR/).find());
+ }
+
+ @Test
+ public void LogstashFilePipelineTest() {
Review comment:
The PR was updated for removing LogstashFilePipelineTest and using
`$LOGSTASH_HOME `instead of a hard-coded path.
For ubuntu-16.04 build issue, let me look into it on my environment and will
give you feddback later, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620:
URL: https://github.com/apache/bigtop/pull/620#discussion_r442598738
##
File path: bigtop-tests/smoke-tests/logstash/TestLogstashSmoke.groovy
##
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.junit.BeforeClass
+import org.junit.AfterClass
+import org.apache.bigtop.itest.shell.Shell
+import static org.junit.Assert.assertTrue
+import static org.junit.Assert.assertFalse
+import org.junit.Test
+import org.apache.bigtop.itest.TestUtils
+import org.junit.runner.RunWith
+import static org.junit.Assert.assertEquals
+
+class TestLogstashSmoke {
+ static Shell sh = new Shell("/bin/bash -s");
+
+ static final String LOGSTASH_CMD = "/usr/lib/logstash/bin/logstash "
+ static final String TINYBENCH_CONFIG = "resources/generator.conf "
+ static final String FILEPIPELINE_CONFIG = "resources/filepipeline.conf "
+
+ @AfterClass
+ public static void LogstashCleanUp() {
+sh.exec("rm -f resources/output.log");
+assertTrue("Logstash cleanup failed. ", sh.getRet() == 0);
+ }
+
+ @Test
+ public void LogstashTinyBenchTest() {
+/* Tiny benchmark with logstash Generator-plugin */
+sh.exec("time "
+ + LOGSTASH_CMD
+ + "-f "
+ + TINYBENCH_CONFIG
+);
+def out = sh.out.join('\n');
+assertFalse("Logstash tiny benchmark failed", (out =~ /ERROR/).find());
+ }
+
+ @Test
+ public void LogstashFilePipelineTest() {
Review comment:
Yes, it makes sense. I'll update it soon, thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620:
URL: https://github.com/apache/bigtop/pull/620#discussion_r428560730
##
File path: bigtop-tests/smoke-tests/logstash/TestLogstashSmoke.groovy
##
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.junit.BeforeClass
+import org.junit.AfterClass
+import org.apache.bigtop.itest.shell.Shell
+import static org.junit.Assert.assertTrue
+import static org.junit.Assert.assertFalse
+import org.junit.Test
+import org.apache.bigtop.itest.TestUtils
+import org.junit.runner.RunWith
+import static org.junit.Assert.assertEquals
+
+class TestLogstashSmoke {
+ static Shell sh = new Shell("/bin/bash -s");
+
+ static final String LOGSTASH_CMD = "/usr/lib/logstash/bin/logstash "
+ static final String TINYBENCH_CONFIG = "resources/generator.conf "
+ static final String FILEPIPELINE_CONFIG = "resources/filepipeline.conf "
+
+ @AfterClass
+ public static void LogstashCleanUp() {
+sh.exec("rm -f resources/output.log");
+assertTrue("Logstash cleanup failed. ", sh.getRet() == 0);
+ }
+
+ @Test
+ public void LogstashTinyBenchTest() {
+/* Tiny benchmark with logstash Generator-plugin */
+sh.exec("time "
+ + LOGSTASH_CMD
+ + "-f "
+ + TINYBENCH_CONFIG
+);
+def out = sh.out.join('\n');
+assertFalse("Logstash tiny benchmark failed", (out =~ /ERROR/).find());
+ }
+
+ @Test
+ public void LogstashFilePipelineTest() {
+def FILE_EXPECTED = 'resources/output-expected.log';
+def FILE_OUTPUT = 'resources/output.log';
+
+sh.exec(LOGSTASH_CMD
+ + "-f "
+ + FILEPIPELINE_CONFIG
+ + "&"
+);
+
+// Wait for completion
+sleep(5);
Review comment:
That's informative, thanks!
The `flush_interval => 0` makes the waiting interval reduce to 5 seconds on
x86.
I will update this PR after verifying it on Arm64.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620:
URL: https://github.com/apache/bigtop/pull/620#discussion_r426352629
##
File path: bigtop-packages/src/common/logstash/do-component-build
##
@@ -0,0 +1,61 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -ex
+
+. `dirname $0`/bigtop.bom
+
+export RELEASE=1
+export LANG="en_US.UTF-8"
+
+if [ -f /etc/os-release ]; then
+. /etc/os-release
+fi
+
+case ${ID} in
+ubuntu | debian)
+sudo apt-get -y update
+sudo apt-get install -y gnupg2
Review comment:
Before installing rvm, we should install GPG keys with `gpg2`. (guide
from https://rvm.io/ )
In ubuntu/deian, `gpg2` is not pre-installed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620:
URL: https://github.com/apache/bigtop/pull/620#discussion_r426414077
##
File path: bigtop-packages/src/common/logstash/do-component-build
##
@@ -0,0 +1,61 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -ex
+
+. `dirname $0`/bigtop.bom
+
+export RELEASE=1
+export LANG="en_US.UTF-8"
+
+if [ -f /etc/os-release ]; then
+. /etc/os-release
+fi
+
+case ${ID} in
+ubuntu | debian)
+sudo apt-get -y update
+sudo apt-get install -y gnupg2
+;;
+esac
+
+case ${ID}-${VERSION_ID} in
+ubuntu-18* | debian-9)
+gpg2 --list-keys
+echo "disable-ipv6" >> ~/.gnupg/dirmngr.conf
+;;
+esac
+
+gpg2 --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
7D2BAF1CF37B13E2069D6956105BD0E739499BDB
+curl -sSL https://get.rvm.io | bash -s stable
+[ -s /usr/local/rvm/scripts/rvm ] && source /usr/local/rvm/scripts/rvm
+
+case ${ID} in
+ubuntu | debian)
+rvm install ruby-2.4.0 && rvm use 2.4.0
+;;
+fedora | centos)
+rvm install ruby-2.3.8 && rvm use 2.3.8
Review comment:
Yes, I agree that all the env specific packages should be installed by
bigtop_toolchain.
But `rvm` and `gpg2` are just used for logstash-package build, and the
bigtop_toolchain is the common facility for all components of Bigtop.
So how about to just install them during `do-components-build` of logstash?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620:
URL: https://github.com/apache/bigtop/pull/620#discussion_r426352664
##
File path: bigtop-packages/src/common/logstash/do-component-build
##
@@ -0,0 +1,61 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -ex
+
+. `dirname $0`/bigtop.bom
+
+export RELEASE=1
+export LANG="en_US.UTF-8"
+
+if [ -f /etc/os-release ]; then
+. /etc/os-release
+fi
+
+case ${ID} in
+ubuntu | debian)
+sudo apt-get -y update
+sudo apt-get install -y gnupg2
+;;
+esac
+
+case ${ID}-${VERSION_ID} in
+ubuntu-18* | debian-9)
+gpg2 --list-keys
+echo "disable-ipv6" >> ~/.gnupg/dirmngr.conf
Review comment:
There are the IPv6 issues happening in docker containers on ubuntu18 and
debian-9:
```
+ gpg2 --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
7D2BAF1CF37B13E2069D6956105BD0E739499BDB
gpg: keyserver receive failed: Cannot assign requested address
```
The soution was provided by rvm.io: https://rvm.io/rvm/security#ipv6-issues
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620:
URL: https://github.com/apache/bigtop/pull/620#discussion_r426352629
##
File path: bigtop-packages/src/common/logstash/do-component-build
##
@@ -0,0 +1,61 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -ex
+
+. `dirname $0`/bigtop.bom
+
+export RELEASE=1
+export LANG="en_US.UTF-8"
+
+if [ -f /etc/os-release ]; then
+. /etc/os-release
+fi
+
+case ${ID} in
+ubuntu | debian)
+sudo apt-get -y update
+sudo apt-get install -y gnupg2
Review comment:
Before installing rvm, we should install GPG keys with `gpg2`. (guide
from https://rvm.io/ )
In ubuntu/deian, `gpg2` is not pre-installed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620: URL: https://github.com/apache/bigtop/pull/620#discussion_r426342373 ## File path: bigtop-packages/src/deb/logstash/logstash.preinst ## @@ -0,0 +1,65 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# preinst script for ignite-hadoop +# +# see: dh_installdeb(1) + +set -e + +# summary of how this script can be called: +#* `install' +#* `install' +#* `upgrade' +#* `abort-upgrade' +# for details, see http://www.debian.org/doc/debian-policy/ or +# the debian-policy package + +LS_USER="logstash" +LS_GROUP="logstash" + +case "$1" in +install|upgrade) +getent group "$LS_GROUP" >/dev/null || groupadd -r "$LS_GROUP" +if ! getent passwd "$LS_USER" >/dev/null; then +# Adding system user: logstash +adduser \ + --system \ + --ingroup "$LS_GROUP" \ + --home /usr/lib/logstash \ + --gecos "Logstash" \ + --disabled-password \ + --shell /bin/false \ + "$LS_USER" >/dev/null Review comment: I missed it and will fix it, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620:
URL: https://github.com/apache/bigtop/pull/620#discussion_r426342107
##
File path: bigtop-tests/smoke-tests/logstash/TestLogstashSmoke.groovy
##
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.junit.BeforeClass
+import org.junit.AfterClass
+import org.apache.bigtop.itest.shell.Shell
+import static org.junit.Assert.assertTrue
+import static org.junit.Assert.assertFalse
+import org.junit.Test
+import org.apache.bigtop.itest.TestUtils
+import org.junit.runner.RunWith
+import static org.junit.Assert.assertEquals
+
+class TestLogstashSmoke {
+ static Shell sh = new Shell("/bin/bash -s");
+
+ static final String LOGSTASH_CMD = "/usr/lib/logstash/bin/logstash "
+ static final String TINYBENCH_CONFIG = "resources/generator.conf "
+ static final String FILEPIPELINE_CONFIG = "resources/filepipeline.conf "
+
+ @AfterClass
+ public static void LogstashCleanUp() {
+sh.exec("rm -f resources/output.log");
+assertTrue("Logstash cleanup failed. ", sh.getRet() == 0);
+ }
+
+ @Test
+ public void LogstashTinyBenchTest() {
+/* Tiny benchmark with logstash Generator-plugin */
+sh.exec("time "
+ + LOGSTASH_CMD
+ + "-f "
+ + TINYBENCH_CONFIG
+);
+def out = sh.out.join('\n');
+assertFalse("Logstash tiny benchmark failed", (out =~ /ERROR/).find());
+ }
+
+ @Test
+ public void LogstashFilePipelineTest() {
+def FILE_EXPECTED = 'resources/output-expected.log';
+def FILE_OUTPUT = 'resources/output.log';
+
+sh.exec(LOGSTASH_CMD
+ + "-f "
+ + FILEPIPELINE_CONFIG
+ + "&"
+);
+
+// Wait for completion
+sleep(5);
Review comment:
This `FILEPIPELINE ` operation would be performed in the background and
output a result file.
It takes time ranging from 20~40 seconds on different platform hosts
(x86/Arm64) from different vendors.
Maybe there are other synchronization mechanisms for this operation but i'm
sorry i don't know. 50 seconds may be a safe estimate to wait for the operation
to complete which is performed asynchronously.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620: URL: https://github.com/apache/bigtop/pull/620#discussion_r415246697 ## File path: bigtop-packages/src/common/logstash/do-component-build ## @@ -0,0 +1,26 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -ex + +. `dirname $0`/bigtop.bom + +export RELEASE=1 +export LANG="en_US.UTF-8" + +rake bootstrap Review comment: The rake would be installed when ruby was installed in bigtop toolchain. `include bigtop_toolchain::ruby` It seems we should rebuild the provisioner docker image to add ruby installation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620: URL: https://github.com/apache/bigtop/pull/620#discussion_r414237239 ## File path: bigtop-packages/src/common/logstash/patch0-logstash.diff ## @@ -0,0 +1,2393 @@ +diff -ruaN logstash-5.4.1/bin/bundle linaro-logstash-5.4.1/bin/bundle Review comment: @evans-ye Yes please, thanks a lot! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component URL: https://github.com/apache/bigtop/pull/620#discussion_r406571789 ## File path: bigtop-packages/src/common/logstash/patch0-logstash.diff ## @@ -0,0 +1,2393 @@ +diff -ruaN logstash-5.4.1/bin/bundle linaro-logstash-5.4.1/bin/bundle Review comment: Yes, it maskes sense. Updated, thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component URL: https://github.com/apache/bigtop/pull/620#discussion_r405243594 ## File path: bigtop-packages/src/common/logstash/do-component-build ## @@ -0,0 +1,27 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -ex + +. `dirname $0`/bigtop.bom + +export RELEASE=1 +export LANG="en_US.UTF-8" +export JAVA_HOME=`readlink -f /usr/bin/java | sed "s:jre/bin/java::"` Review comment: Agree to leverage Bigtop script to detect JAVA_HOME, thanks for the info. But I found that it's unnecessary to detect JAVA_HOME in `do-component-build` , and in the phase of `do-component-build` , `bigtop-utils` has not be ready yet. Removed the JAVA_HOME detection. Thanks for the comments, envans! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component URL: https://github.com/apache/bigtop/pull/620#discussion_r405243594 ## File path: bigtop-packages/src/common/logstash/do-component-build ## @@ -0,0 +1,27 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -ex + +. `dirname $0`/bigtop.bom + +export RELEASE=1 +export LANG="en_US.UTF-8" +export JAVA_HOME=`readlink -f /usr/bin/java | sed "s:jre/bin/java::"` Review comment: Agree to leverage Bigtop script to detect JAVA_HOME, thanks for the info. But I found that it's unnecessary to detect JAVA_HOME in `do-component-build` , and in the phase of `do-component-build` , `bigtop-utils` has not be ready yet. Removed the JAVA_HOME detection. Thanks for the comments, evans! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as
Bigtop component
URL: https://github.com/apache/bigtop/pull/620#discussion_r405239731
##
File path: bigtop_toolchain/manifests/ruby.pp
##
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+class bigtop_toolchain::ruby {
+
+ require bigtop_toolchain::packages
+
+ $url = "https://cache.ruby-lang.org/pub/ruby/2.4;
+
+ $ruby24 = "ruby-2.4.0.tar.gz"
+ $ruby24dir = "ruby-2.4.0"
+
+ exec { "download ruby":
+ cwd => "/usr/src",
+ command => "/usr/bin/wget $url/$ruby24 && mkdir -p $ruby24dir && /bin/tar
-xvzf $ruby24 -C $ruby24dir --strip-components=1 && cd $ruby24dir",
+ creates => "/usr/src/$ruby24dir",
+ }
+
+ exec { "install ruby":
+ cwd => "/usr/src/$ruby24dir",
+ command => "/usr/src/$ruby24dir/configure --prefix=/usr/local
--disable-shared && /usr/bin/make install",
+ creates => "/usr/local/bin/ruby",
+ timeout => 1800,
+ require => EXEC["download ruby"]
+ }
Review comment:
Yea, for the dependency issues, the ruby-2.4 is the latest version that
works with logstash-5.4.1.
And the bigtop should support different kind of distros and
platforms(x86/arm64).
It may not be guaranteed the various distros could offer the ruby-2.4 pkg
(x86/arm64) installation. IMHO, it's robust to build the ruby-2.4 binaries from
source though it may cost extra time in bigtop_toolchain build.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as
Bigtop component
URL: https://github.com/apache/bigtop/pull/620#discussion_r405239731
##
File path: bigtop_toolchain/manifests/ruby.pp
##
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+class bigtop_toolchain::ruby {
+
+ require bigtop_toolchain::packages
+
+ $url = "https://cache.ruby-lang.org/pub/ruby/2.4;
+
+ $ruby24 = "ruby-2.4.0.tar.gz"
+ $ruby24dir = "ruby-2.4.0"
+
+ exec { "download ruby":
+ cwd => "/usr/src",
+ command => "/usr/bin/wget $url/$ruby24 && mkdir -p $ruby24dir && /bin/tar
-xvzf $ruby24 -C $ruby24dir --strip-components=1 && cd $ruby24dir",
+ creates => "/usr/src/$ruby24dir",
+ }
+
+ exec { "install ruby":
+ cwd => "/usr/src/$ruby24dir",
+ command => "/usr/src/$ruby24dir/configure --prefix=/usr/local
--disable-shared && /usr/bin/make install",
+ creates => "/usr/local/bin/ruby",
+ timeout => 1800,
+ require => EXEC["download ruby"]
+ }
Review comment:
Yea, for the dependency issues, the ruby-2.4 is the latest version that that
works with logstash-5.4.1.
And the bigtop should support different kind of distros and
platforms(x86/arm64).
It may not be guaranteed the various distros could offer the ruby-2.4 pkg
(x86/arm64) installation. IMHO, it's robust to build the ruby-2.4 binaries from
source though it may cost extra time in bigtop_toolchain build.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [bigtop] guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component
guyuqi commented on a change in pull request #620: BIGTOP-3319: Add Logstash as Bigtop component URL: https://github.com/apache/bigtop/pull/620#discussion_r405234038 ## File path: bigtop-packages/src/common/logstash/patch0-logstash.diff ## @@ -0,0 +1,2393 @@ +diff -ruaN logstash-5.4.1/bin/bundle linaro-logstash-5.4.1/bin/bundle Review comment: Thanks for comment, evans. 1. Pick the logstash version: 5.4.1. It corresponds to the elasticsearch version in Bigtop. 2. `patch0-logstash.diff` is to support Arm64. The Jruby-1.9 in logstash-5.4.1 doesn't support Arm64 due to the fact that Jffi 1.2.12 which Jruby 1.9 depends on has no native AArch64 libraries. Also see: [jffi](https://github.com/jnr/jffi/pull/32), [jruby1](https://github.com/jruby/jruby/pull/4548), [jruby2](https://github.com/jruby/jruby/pull/4567). So `patch0-logstash.diff` upgraded jruby to 9K and also fix dependency issues for Arm64. The modification for Arm64 support was archived in [linaro repo](https://git.linaro.org/leg/bigdata/logstash.git) and `patch0-logstash.diff` comes from it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
