oleewere closed pull request #68: AMBARI-23456. Add developement guide 
markdowns & remove some old docs
URL: https://github.com/apache/ambari-logsearch/pull/68
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/README.md b/README.md
index 21b94fc378..142587e6a9 100644
--- a/README.md
+++ b/README.md
@@ -2,89 +2,38 @@
 [![Build 
Status](https://builds.apache.org/buildStatus/icon?job=Ambari-LogSearch-master-Commit)](https://builds.apache.org/view/A/view/Ambari/job/Ambari-LogSearch-master-Commit/)
 ![license](http://img.shields.io/badge/license-Apache%20v2-blue.svg)
 
-Log aggregation, analysis, and visualization for Ambari managed (and any 
other) services.
-
 Log Search is a sub-project of [Apache 
Ambari](https://github.com/apache/ambari)
 
-## Development
-
-Requires JDK 8 (JDK 11 is recommended) and Maven 3.5.x
-
-### Prerequisites
-
-- Install [docker](https://docs.docker.com/)
-- For Mac OS X use [Docker for Mac](https://docs.docker.com/docker-for-mac/)
-- [Docker compose](https://docs.docker.com/compose/) is also required.
-
-### Build and start Log Search in docker container
-```bash
-# to see available commands: run start-logsearch without arguments
-cd docker
-./logsearch-docker build-and-run # build mvn project locally, build docker 
image, start containers
-```
-If you run the script at first time, it will generate you a new `Profile` file 
or an `.env` file inside docker directory (run twice if both missing and you 
want to generate Profile and .env as well), in .env file you should set 
`MAVEN_REPOSITORY_LOCATION` (point to local maven repository location, it uses 
`~/.m2` by default). These will be used as volumes for the docker container. 
Profile file holds the environment variables that are used inside the 
containers, the .env file is used outside of the containers
-
-Then you can use the `logsearch-docker` script to start the containers 
(`start` command).
-Also you can use docker-compose manually to start/manage the containers.
-```bash
-docker-compose up -d
-# or start all services in one container:
-docker-compose -f all.yml up -d
-```
-After the logsearch container is started you can enter to it with following 
commands:
-```bash
-docker exec -it docker_logsearch_1 bash
-# or if you used all.yml for starting the logsearch docker container:
-docker exec -it logsearch bash
-```
-In case if you started the containers separately and if you would like to 
access Solr locally with through your external ZooKeeper container, then point 
`solr` to `localhost` in your `/etc/hosts` file.
-
-### Run applications from IDE / maven
-
-- [Start Log Search locally](ambari-logsearch-server/README.md)
-- [Start Log Feeder locally](ambari-logsearch-logfeeder/README.md)
-
-## Package build process
+## Introduction
 
-1. Check out the code from GIT repository
+Log aggregation, analysis, and visualization for Ambari managed (or any other) 
services.
 
-2. On the logsearch root folder (ambari/ambari-logsearch), please execute the 
following make command to build RPM/DPKG:
-```bash
-make rpm
-# or for jdk11
-export LOGSEARCH_JDK_11=true
-make rpm
-```
-  or
-```bash
-make deb
-# or for jdk11
-export LOGSEARCH_JDK_11=true
-make deb
-```
-3. Generated RPM/DPKG files will be found in ambari-logsearch-assembly/target 
folder
+## Features
 
-## Running Integration Tests
+- Parse / aggregate and ship logs
+- Send and index logs in Solr
+- Store logs (structured or un-sructured format) in Cloud Storage (S3 / GCS / 
ADLS / WASB)
+- Fultext Search in logs (if the logs are shipped to Solr)
+- JWT/SSO support
+- Support testing the log parsing on the UI
 
-By default integration tests are not a part of the build process, you need to 
set -Dbackend-tests or -Dselenium-tests (or you can use -Dall-tests to run 
both). To running the tests you will need docker here as well (right now 
docker-for-mac and unix are supported by default, for boot2docker you need to 
pass -Ddocker.host parameter to the build).
+## Architecture
 
-```bash
-# from ambari-logsearch folder
-mvn clean integration-test -Dbackend-tests failsafe:verify
-# or run selenium tests with docker for mac, but before that you nedd to start 
xquartz
-open -a XQuartz
-# then in an another window you can start ui tests
-mvn clean integration-test -Dselenium-tests failsafe:verify
-# you can specify story file folde location with -Dbackend.stories.location 
and -Dui.stories.location (absolute file path) in the commands
-```
-Also you can run from the IDE, but make sure all of the ambari logsearch 
modules are built.
+- Log Feeder: agent component on all hosts to monitor and shipping logs.
+- Log Search Portal: REST API + UI for rendering logs
+- Solr (Optional - default): storage for logs, used by both Log Search Portal 
and Log Feeder
+- ZooKeeper (Optional - default): configuration service for Solr, Log Search 
and Log Feeder
+- HDFS / S3 / GCS / ADLS: storage for logs (write only), used by Log Feeder 
[cloud mode](docs/cloud_mode.md)
 
+![Log Search Architecture Overview](docs/images/architecture_overview.jpg)
 
-## Update version (for release or specific builds)
+## Contents
 
-```bash
-make update-version new-version="2.8.0.0-11"
-```
+- [1, Installation](docs/installation.md)
+- [2. Collections](docs/collections.md)
+- [3. Adding new logs to monitor](docs/add_new_input.md) 
+- [4. Development guide](docs/development.md)
+- [5. Using Log Feeder in Cloud mode](docs/cloud_mode.md)
 
 ## Contributing
 
diff --git a/ambari-logsearch-logfeeder/README.md 
b/ambari-logsearch-logfeeder/README.md
deleted file mode 100644
index 48cb81b49b..0000000000
--- a/ambari-logsearch-logfeeder/README.md
+++ /dev/null
@@ -1,49 +0,0 @@
-<!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements.  See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License.  You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-# Log Search - Log Feeder:
-
-Log Feeder is a component of the Log Search service that reads logs, parses 
them and stores them in Apache Solr for the purpose
-of later analysis.
-
-## Start locally from maven / IDE
-
-First you need to start every required service (except logfeeder), go to 
`ambari-logsearch/docker` folder and run:
-```bash
-docker-compose up -d zookeeper solr logsearch
-```
-
-Secondly, if you are planning to run Log Feeder from an IDE, for running the 
LogFeeder main methoud, you will need to set the working directory to 
`ambari/ambari-logsearch/ambari-logsearch-logfeeder` or set 
`LOGFEEDER_RELATIVE_LOCATION` env variable.
-With Maven, you won't need these steps, just run this command from the 
ambari-logsearch-logfeeder folder:
-
-```bash
-mvn clean package -DskipTests spring-boot:run
-```
-
-# Input Configuration
-
-The configuration for the log feeder contains
-* description of the log files
-* description of the filters that parse the data of the log entries
-* description of the mappers that modify the parsed fields
-
-The element description can be found [here](docs/inputConfig.md)
-
-All these data are stored in json files, which should be named in the 
directory /etc/ambari-logsearch-logfeeder/conf, and the
-name of the files should be input.config-<service\_name>.json
\ No newline at end of file
diff --git a/ambari-logsearch-server/README.md 
b/ambari-logsearch-server/README.md
deleted file mode 100644
index 7432344dc7..0000000000
--- a/ambari-logsearch-server/README.md
+++ /dev/null
@@ -1,39 +0,0 @@
-<!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements.  See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License.  You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-# Log Search Server
-
-## Start locally from maven / IDE
-
-Other services (like zookeeper, solr, logfeeder) can be started with 
`docker-compose`
-```bash
-cd ambari/ambari-logsearch/docker
-docker-compose up -d zookeeper solr logfeeder
-```
-
-Then you can start Log Search server from maven 
-
-```bash
-cd ambari/ambari-logsearch/ambari-logsearch-server
-./run.sh
-# or
-mvn clean package -DskipTests spring-boot:run
-```
-
-You can also start Log Search server from an IDE as well. One thing is 
important: the config set location that the server tries to upload to 
ZooKeeper. By default config sets are located at 
`${LOGSEARCH_SERVER_RELATIVE_LOCATION:}src/main/configsets` in 
`logsearch.properties`. Based or from where you run `LogSearch.java`, you need 
to set `LOGSEARCH_SERVER_RELATIVE_LOCATION` env variable properly. 
diff --git a/docs/development.md b/docs/development.md
index 9a27bef0f1..866d435b6e 100644
--- a/docs/development.md
+++ b/docs/development.md
@@ -15,9 +15,124 @@ See the License for the specific language governing 
permissions and
 limitations under the License.
 -->
 
-### Backend development
+## Developement guide
 
-TODO
+## Requirements
+
+- JDK 8 (JDK 11 is recommended) 
+- Maven 3.5.x
+
+## Backend development
+
+### Prerequisites
+
+- Install [docker](https://docs.docker.com/)
+- For Mac OS X use [Docker for Mac](https://docs.docker.com/docker-for-mac/)
+- [Docker compose](https://docs.docker.com/compose/) is also required.
+
+### Build and start Log Search in docker container
+```bash
+# to see available commands: run start-logsearch without arguments
+cd docker
+./logsearch-docker build-and-run # build mvn project locally, build docker 
image, start containers
+```
+If you run the script at first time, it will generate you a new `Profile` file 
or an `.env` file inside docker directory (run twice if both missing and you 
want to generate Profile and .env as well), in .env file you should set 
`MAVEN_REPOSITORY_LOCATION` (point to local maven repository location, it uses 
`~/.m2` by default). These will be used as volumes for the docker container. 
Profile file holds the environment variables that are used inside the 
containers, the .env file is used outside of the containers
+
+Then you can use the `logsearch-docker` script to start the containers 
(`start` command).
+Also you can use docker-compose manually to start/manage the containers.
+```bash
+docker-compose up -d
+```
+After the logsearch container is started you can enter to it with following 
commands:
+```bash
+docker exec -it docker_logsearch_1 bash
+```
+In case if you started the containers separately and if you would like to 
access Solr locally with through your external ZooKeeper container, then point 
`solr` to `localhost` in your `/etc/hosts` file.
+
+### Enable debug for Log Search Server or Log Feeder
+
+By default remote debug is enabled in docker containers for Log Search server 
and Log Feeder (in order to use IDEs for debugging). Log Search server port for 
debug is `5005` and `5006` for Log Feeder. 
+
+To suspend components in debug mode, you need to edit your `Profile` file 
(that was generated in docker folder) and set `LOGSEARCH_DEBUG_SUSPEND` or 
`LOGFEEDER_DEBUG_SUSPEND` to `true`.
+
+### Run applications from IDE / maven
+
+#### Start Log Search server locally from maven / IDE
+
+Other services (like zookeeper, solr, logfeeder) can be started with 
`docker-compose`
+```bash
+cd ambari/ambari-logsearch/docker
+docker-compose up -d zookeeper solr logfeeder
+```
+
+Then you can start Log Search server from maven 
+
+```bash
+cd ambari/ambari-logsearch/ambari-logsearch-server
+./run.sh
+# or
+mvn clean package -DskipTests spring-boot:run
+```
+
+You can also start Log Search server from an IDE as well. One thing is 
important: the config set location that the server tries to upload to 
ZooKeeper. By default config sets are located at 
`${LOGSEARCH_SERVER_RELATIVE_LOCATION:}src/main/configsets` in 
`logsearch.properties`. Based or from where you run `LogSearch.java`, you need 
to set `LOGSEARCH_SERVER_RELATIVE_LOCATION` env variable properly. (or just 
simply use the ambari-logsearch-server as the working directory)
+
+#### Start Log Feeder locally from maven / IDE
+
+First you need to start every required service (except logfeeder), go to 
`ambari-logsearch/docker` folder and run:
+```bash
+docker-compose up -d zookeeper solr logsearch
+```
+
+Secondly, if you are planning to run Log Feeder from an IDE, for running the 
LogFeeder main methoud, you will need to set the working directory to 
`ambari/ambari-logsearch/ambari-logsearch-logfeeder` or set 
`LOGFEEDER_RELATIVE_LOCATION` env variable.
+With Maven, you won't need these steps, just run this command from the 
ambari-logsearch-logfeeder folder:
+
+```bash
+mvn clean package -DskipTests spring-boot:run
+```
+
+For Log Feeder, it is also important to use the ambari-logsearch-logfeeder as 
a working directory if you are trying to run the application from an IDE.
+
+### Package build process
+
+1. Check out the code from GIT repository
+
+2. On the logsearch root folder (ambari/ambari-logsearch), please execute the 
following make command to build RPM/DPKG:
+```bash
+make rpm
+# or for jdk11
+export LOGSEARCH_JDK_11=true
+make rpm
+```
+  or
+```bash
+make deb
+# or for jdk11
+export LOGSEARCH_JDK_11=true
+make deb
+```
+3. Generated RPM/DPKG files will be found in ambari-logsearch-assembly/target 
folder
+
+### Running Integration Tests
+
+By default integration tests are not a part of the build process, you need to 
set -Dbackend-tests or -Dselenium-tests (or you can use -Dall-tests to run 
both). To running the tests you will need docker here as well (right now 
docker-for-mac and unix are supported by default, for boot2docker you need to 
pass -Ddocker.host parameter to the build).
+
+```bash
+# from ambari-logsearch folder
+mvn clean integration-test -Dbackend-tests failsafe:verify
+# or run selenium tests with docker for mac, but before that you nedd to start 
xquartz
+open -a XQuartz
+# then in an another window you can start ui tests
+mvn clean integration-test -Dselenium-tests failsafe:verify
+# you can specify story file folde location with -Dbackend.stories.location 
and -Dui.stories.location (absolute file path) in the commands
+```
+Also you can run from the IDE, but make sure all of the ambari logsearch 
modules are built.
+
+
+### Update version (for release or specific builds)
+
+```bash
+make update-version new-version="2.8.0.0-11"
+```
 
 ### UI development
 


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Reply via email to