This is an automated email from the ASF dual-hosted git repository.
oleewere pushed a commit to branch branch-feature-AMBARI-23456
in repository https://gitbox.apache.org/repos/asf/ambari-logsearch.git
commit b2c1602b0dda3e999d4db5735edd300e37483b64
Author: Oliver Szabo <oleew...@gmail.com>
AuthorDate: Tue Dec 11 02:29:23 2018 +0100
AMBARI-23456. Add more markdown docs
---
ambari-logsearch-docs/pom.xml | 1 +
docs/about.md | 12 ++-
docs/add_new_input.md | 94 ++++++++++++++++++-
docs/cloud_mode.md | 6 +-
docs/collections.md | 62 +++++++++++++
docs/{getting_started.md => development.md} | 8 +-
docs/docs.md | 14 ++-
docs/images/add_service.png | Bin 0 -> 282737 bytes
docs/images/add_service_wizard.png | Bin 0 -> 278205 bytes
docs/images/architecture_overview.jpg | Bin 0 -> 453385 bytes
docs/images/edit_configs.jpg | Bin 0 -> 189354 bytes
docs/images/test_sample.jpg | Bin 0 -> 214507 bytes
docs/images/test_sample_result.jpg | Bin 0 -> 260567 bytes
docs/index.md | 29 +++++-
docs/installation.md | 138 ++++++++++++++++++++++++++++
docs/schema_fields.md | 16 ----
mkdocs.yml | 11 ++-
pom.xml | 1 +
18 files changed, 363 insertions(+), 29 deletions(-)
diff --git a/ambari-logsearch-docs/pom.xml b/ambari-logsearch-docs/pom.xml
index eefdc6b..90a19a9 100644
--- a/ambari-logsearch-docs/pom.xml
+++ b/ambari-logsearch-docs/pom.xml
@@ -28,6 +28,7 @@
<packaging>jar</packaging>
<url>http://maven.apache.org</url>
<name>Ambari Logsearch Docs</name>
+ <description>Ambari Logsearch Docs</description>
<build>
<resources>
<resource>
diff --git a/docs/about.md b/docs/about.md
index 4fb1258..11625d6 100644
--- a/docs/about.md
+++ b/docs/about.md
@@ -13,4 +13,14 @@ distributed under the License is distributed on an "AS IS"
BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--->
\ No newline at end of file
+-->
+
+## Contribution Guide
+
+https://cwiki.apache.org/confluence/display/AMBARI/How+to+Contribute
+
+(That is the ambari contribution guide, everything is the same here except use
ambari-logsearch repository instead of ambari)
+
+## License
+
+[Apache 2.0 LICENSE](https://www.apache.org/licenses/LICENSE-2.0)
\ No newline at end of file
diff --git a/docs/add_new_input.md b/docs/add_new_input.md
index 4fb1258..24c3e30 100644
--- a/docs/add_new_input.md
+++ b/docs/add_new_input.md
@@ -13,4 +13,96 @@ distributed under the License is distributed on an "AS IS"
BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--->
\ No newline at end of file
+-->
+
+## Adding New Logs to Monitor
+
+#### Input / Filter (shipper) descriptors
+
+For adding a new log that will be monitored by Log Feeder, first, it is needed
to understand how Log Feeder registers and use those logs as inputs.
+All the input log descriptors are stored in Log Feeder configuration directory
with `input.config-<service>.json` naming pattern. (see: [Shipper
configurations](shipper_configurations.md))
+
+Based on these descriptors, the Log Feeder will start to monitor the inputs
that are defined inside the descriptor as inputs.
+After the configs are first processed, those will be handled/uploaded by a
configuration handler, that should be an implementation of the Config API.
+
+#### Config API
+
+The Config API is responsible for to load and handle input (and filter)
descriptors for Log Feeder. The idea is to store those configurations in a
shared system
+and Log Feeders should be notified about any of the input configuration
changes.
+It has different implementations, the default one is ZooKeeper (there is 2
other one right now: Solr and Local as well). Also with using/enabling the
Config API,
+the user will be able to edit / test / add input descriptors.
+
+#### Support new input descriptor(s) for Ambari components
+
+- Create a new jinja template:
`<SERVICE_LOCATION>/package/templates/input.config-<service_name>.json.j2`
+
+```jinja2
+{
+ "input":[
+ {
+ "type":"my_log",
+ "rowtype":"service",
+ "path":"{{default('/configurations/myservice-env/log_dir',
'/var/log/myservice')}}/my_log*.log"
+ },
+ {
+ "type":"myother_log",
+ "rowtype":"service",
+ "path":"{{default('/configurations/myservice-env/log_dir',
'/var/log/myservice')}}/myother_log*.log"
+ }
+ ],
+ "filter": [
+ ...
+ ]
+```
+
+- Change the stack code to write the input descriptor (add this to the start
script):
+
+```python
+from resource_management.core.source import Template
+from resource_management.libraries.functions.default import default
+from resource_management.libraries.functions.generate_logfeeder_input_config
import generate_logfeeder_input_config
+
+# ...
+ # that will write input.config.myservice.json file to
/usr/lib/ambari-logsearch-logfeeder/conf folder
+ generate_logfeeder_input_config('myservice',
Template("input.config-myservice.json.j2", extra_imports=[default]))
+# ...
+```
+- Edit the `metainfo.xml`: add logId mapping for components (logIds are log
types in input descriptors)
+```xml
+<!-- ... -->
+ <components>
+ <component>
+ <name>MYCOMPONENT</name>
+ <logs>
+ <log>
+ <logId>my_log</logId>
+ <primary>true</primary>
+ </log>
+ <log>
+ <logId>myother_log</logId>
+ </log>
+ </logs>
+ <!-- ... -->
+ <component>
+ <components>
+ <!-- ... -->
+```
+From that point - during service start - the input config desriptor will be
written into `/usr/lib/ambari-logsearch-logfeeder/conf`. The Log Feeder
applications monitor the files in that folder, if there is a new one, it will
start to monitor those inputs. Note that if the Config API is configured to use
ZooKeeper, these configurations are only used from bootstrapping, after these
files are uploaded to ZooKeeper, you will be onlyable to edit the input
descriptos by Log Search Portal (or [...]
+
+#### Add / Edit / Test input desriptors by Log Search Portal
+
+On the Log Search Portal (if you go into `Configuration Editor`) you can edit
(or add) existing input / filter configurations:
+
+
+
+Also you can add test samples that you can validate:
+
+
+
+If the validation is successful, you should got a map response with the
created fields:
+
+
+
+### Add input descriptors without Ambari
+
+As you need the `input.config-<service>.json` files in
`/usr/lib/ambari-logsearch-logfeeder/conf` folder (or where your configuration
is located), it is enough to just create a new file at that location.
\ No newline at end of file
diff --git a/docs/cloud_mode.md b/docs/cloud_mode.md
index 4fb1258..731c6d2 100644
--- a/docs/cloud_mode.md
+++ b/docs/cloud_mode.md
@@ -13,4 +13,8 @@ distributed under the License is distributed on an "AS IS"
BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--->
\ No newline at end of file
+-->
+
+### Log Feeder: cloud mode
+
+TODO
\ No newline at end of file
diff --git a/docs/collections.md b/docs/collections.md
new file mode 100644
index 0000000..7031cad
--- /dev/null
+++ b/docs/collections.md
@@ -0,0 +1,62 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+### Collections
+
+By default there are 3 main storage layer abstractions in Log Search:
+
+- Service logs (default name: `hadoop_logs`)
+- Audit logs (default name: `audit_logs`)
+- Metadata (default name: `logsearch_metadata`)
+
+Service logs collection is responsible to store most of the logs (by type),
except audit related data, for those use the audit logs collection.
+The metadata collection is used to store Log Search UI related (dynamic)
configurations/settings.
+
+### Schema fields
+
+Minimal required service log fields in Solr to make it work with the UI:
+
+- id (string, unique - identifier for Solr doc)
+- log_message
+- type (string - log type)
+- logtime (timestamp)
+- seq_num (numeric - sequence number for log events, useful to not sort only
by date)
+- level (string - log level for logs, e.g.: WARN)
+- host (string)
+- cluster (string)
+
+see more: [Service logs
schema](https://github.com/apache/ambari-logsearch/blob/master/ambari-logsearch-server/src/main/configsets/hadoop_logs/conf/managed-schema)
+
+Minimal required audit log fields in Solr to make it work with the UI:
+
+- id (string, unique - identifier for Solr doc)
+- evtTime (timestamp)
+- repo (string - represents the audit source type)
+- seq_num (numeric - sequence number for log events, useful to not sort only
by date)
+
+see more: [Audit logs
schema](https://github.com/apache/ambari-logsearch/blob/master/ambari-logsearch-server/src/main/configsets/audit_logs/conf/managed-schema)
+
+Fields for metadata:
+- id (string, unique - identifier for Solr doc)
+- name (string - metadata identifier)
+- username (string - for identify user related data)
+- type (string - type of the metadata)
+- value (string - can be anything)
+
+### Customize field names on the Log Search UI
+
+Field name labels on the UI can be customized in `logsearch.properties`, see:
[AMBARI-22842](https://issues.apache.org/jira/browse/AMBARI-22842)
\ No newline at end of file
diff --git a/docs/getting_started.md b/docs/development.md
similarity index 93%
rename from docs/getting_started.md
rename to docs/development.md
index f51f3ec..9a27bef 100644
--- a/docs/getting_started.md
+++ b/docs/development.md
@@ -15,8 +15,10 @@ See the License for the specific language governing
permissions and
limitations under the License.
-->
-## Installation
+### Backend development
-## Build
+TODO
-## Requirements
\ No newline at end of file
+### UI development
+
+TODO
\ No newline at end of file
diff --git a/docs/docs.md b/docs/docs.md
index 4fb1258..efd7ebe 100644
--- a/docs/docs.md
+++ b/docs/docs.md
@@ -13,4 +13,16 @@ distributed under the License is distributed on an "AS IS"
BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--->
\ No newline at end of file
+-->
+
+### Rest API doc
+
+Swagger specification:
[logsearch-swagger.yaml](api-docs/logsearch-swagger.yaml)
+
+### Javadoc
+
+Generate javadoc with the following command from the project root:
+
+```bash
+make javadoc
+```
\ No newline at end of file
diff --git a/docs/images/add_service.png b/docs/images/add_service.png
new file mode 100644
index 0000000..fc97f04
Binary files /dev/null and b/docs/images/add_service.png differ
diff --git a/docs/images/add_service_wizard.png
b/docs/images/add_service_wizard.png
new file mode 100644
index 0000000..cf83462
Binary files /dev/null and b/docs/images/add_service_wizard.png differ
diff --git a/docs/images/architecture_overview.jpg
b/docs/images/architecture_overview.jpg
new file mode 100644
index 0000000..476e81f
Binary files /dev/null and b/docs/images/architecture_overview.jpg differ
diff --git a/docs/images/edit_configs.jpg b/docs/images/edit_configs.jpg
new file mode 100644
index 0000000..7599685
Binary files /dev/null and b/docs/images/edit_configs.jpg differ
diff --git a/docs/images/test_sample.jpg b/docs/images/test_sample.jpg
new file mode 100644
index 0000000..1f5e3f3
Binary files /dev/null and b/docs/images/test_sample.jpg differ
diff --git a/docs/images/test_sample_result.jpg
b/docs/images/test_sample_result.jpg
new file mode 100644
index 0000000..ba208a9
Binary files /dev/null and b/docs/images/test_sample_result.jpg differ
diff --git a/docs/index.md b/docs/index.md
index 17fa30e..e994a61 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -18,4 +18,31 @@ limitations under the License.
Log aggregation, analysis, and visualization for Ambari managed (or any other)
services.
-## Architecture
\ No newline at end of file
+## Features
+
+- Parse / aggregate and ship logs
+- Send and index logs in Solr
+- Store logs (structured or un-sructured format) in Cloud Storage (S3 / GCS /
ADLS / WASB)
+- Fultext Search in logs (if the logs are shipped to Solr)
+- JWT/SSO support
+- Support testing the log parsing on the UI
+
+## Architecture
+
+- Log Feeder: agent component on all hosts to monitor and shipping logs.
+- Log Search Portal: REST API + UI for rendering logs
+- Solr (Optional - default): storage for logs, used by both Log Search Portal
and Log Feeder
+- ZooKeeper (Optional - default): configuration service for Solr, Log Search
and Log Feeder
+- HDFS / S3 / GCS / ADLS: storage for logs (write only), used by Log Feeder
[cloud mode](cloud_mode.md)
+
+
+
+## Contents
+
+- [1, Installation](installation.md)
+- [2. Collections](collections.md)
+- [3. Adding new logs to monitor](add_new_input.md)
+- [4. Development guide](development.md)
+- [5. Using Log Feeder in Cloud mode](cloud_mode.md)
+- [6. Contribution Guide and License](about.md)
+
diff --git a/docs/installation.md b/docs/installation.md
new file mode 100644
index 0000000..05dd8d9
--- /dev/null
+++ b/docs/installation.md
@@ -0,0 +1,138 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+## Installation
+
+#### Requirements for building the project
+
+- Java 8+
+- Maven 3.5.x +
+- `rpm-build` on Mac OS X
+
+#### Checkout the project
+
+```bash
+git clone https://github.com/apache/ambari-logsearch
+cd ambari-logsearch
+```
+#### Build Log Search packages
+
+- a.) Create RPM packages (outputs: ambari-logsearch-assembly/target/rpms/*)
+```bash
+make rpm
+```
+
+- b.) Create DEB packages (outputs: ambari-logsearch-assembly/target/*.deb)
+```bash
+make deb
+```
+
+- c.) Create tarballs (outputs:
ambari-logsearch-server/target/ambari-logsearch-portal.tar.gz and
ambari-logsearch-logfeeder/target/ambari-logsearch-logfeeder.tar.gz)
+```bash
+make package
+```
+
+#### Install created packages
+
+- On RHEL / CentOS :
+```bash
+# Log Search Portal (server + UI)
+yum install -y /my/path/ambari-logsearch-portal.rpm
+# Log Feeder
+yum install -y /my/path/ambari-logsearch-logfeeder.rpm
+# or if a repo is available that contains logsearch packages
+yum install -y ambari-logsearch-portal
+yum install -y ambari-logsearch-logfeeder
+```
+- On Debian / Ubuntu :
+```bash
+# Log Search Portal (server + UI)
+dpkg -i /my/path/ambari-logsearch-portal.deb
+# Log Feeder
+dpkg -i /my/path/ambari-logsearch-logfeeder.deb
+# or if a repo is available that contains logsearch packages
+apt-get install ambari-logsearch-portal
+apt-get install ambari-logsearch-logfeeder
+```
+- With tarball :
+```bash
+mkdir logsearch
+tar -xzf ambari-logsearch-portal.tar.gz -C logsearch
+
+mkdir logfeeder
+tar -xzf ambari-logsearch-logfeeder.tar.gz -C logfeeder
+```
+
+#### Install by Ambari UI
+
+Click on Add Service page:
+
+
+
+Then select Log Search on Add Services page:
+
+
+
+Then (after setting the right configuration on Configure services page) you
can start the Log Search / Log Feeder deployments.
+
+#### Component names to use in Ambari Blueprints:
+
+- Log Search Portal: `LOGSEARCH_SERVER` (1 node, or 1+, as Log Search Portal
is stateless)
+- Log Feeder: `LOGSEARCH_LOGFEEDER`(All nodes)
+
+#### Requirements for using Log Search
+
+- `Solr` 6.x+ and `ZooKeeper` (as Solr is used in Cloud mode) is required for
Log Search Portal - Ambari Infra Solr can be used as a built-in Solr service in
Ambari, but of course it can use an external Solr as well.
+- Log Feeder in Cloud mode can work without Solr and ZooKeeper (that means
logs will be shipped to HDFS/S3/GCS/WASB/ADLSv2 ... but Log Search Portal won't
be able to display data from those Storages)
+
+#### Configurations
+
+##### Log Search Portal configuration:
+
+Required configuration files for Log Search Portal at
`/usr/lib/ambari-logsearch-portal/conf`:
+
+- logsearch.properties
+- logsearch-env.sh
+- log4j2.yml
+- info.properties
+- default.properties
+
+##### Log Feeder configuration:
+
+Required configuration files for Log Feeder at
`/usr/lib/ambari-logsearch-logfeeder/conf`:
+
+- logfeeder.properties
+- logfeeder-env.sh
+- log4j2.yml
+- alias_config.json
+- global.config.json
+- output.config.json
+- input.config-<service_name>.json (use this filename pattern for input
configurations)
+- checkponts/*.cp (checkpoint files - information about monitored input files)
+
+
+See more about the configurations: [Log Search
properties](logsearch_properties.md), [Log Feeder
properties](logfeeder_properties.md) and [Log Feeder shipper
configurations](shipper_configurations.md)
+
+#### Starting Log Search / Log Feeders
+
+Go to Ambari UI and start Log Search components or manually starts by
`logsearch.sh` or `logfeeder.sh` script
+```bash
+# starting Log Search
+/usr/lib/ambari-logsearch-portal/bin/logsearch.sh start
+# starting Log Feeder
+/usr/lib/ambari-logsearch-logfeeder/bin/logsearch.sh start
+```
\ No newline at end of file
diff --git a/docs/schema_fields.md b/docs/schema_fields.md
deleted file mode 100644
index 4fb1258..0000000
--- a/docs/schema_fields.md
+++ /dev/null
@@ -1,16 +0,0 @@
-<!---
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
\ No newline at end of file
diff --git a/mkdocs.yml b/mkdocs.yml
index a4d46a5..6a2a7a0 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -15,15 +15,16 @@
site_name: Ambari Log Search
site_dir: mkdocs-site
nav:
-- Home: index.md
+- Overview: index.md
- User Guide:
- - Getting Started: getting_started.md
- - Add New Service: add_new_input.md
- - Schema Fields: schema_fields.md
+ - Installation: installation.md
+ - Collections: collections.md
+ - Add New Logs: add_new_input.md
- Cloud Mode: cloud_mode.md
+ - Development: development.md
- Configuration:
- Log Search properties: logsearch_properties.md
- Log Feeder properties: logfeeder_properties.md
- Shipper configurations: shipper_configurations.md
-- Docs: docs.md
+- API Docs: docs.md
- About: about.md
\ No newline at end of file
diff --git a/pom.xml b/pom.xml
index 6a3157b..c6826b7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -23,6 +23,7 @@
<packaging>pom</packaging>
<name>Ambari Logsearch Parent</name>
<url>http://maven.apache.org</url>
+ <description>Ambari Logsearch Parent</description>
<profiles>
<profile>
<id>all</id>
