oleewere closed pull request #66: AMBARI-23456. Add cloud mode documentation URL: https://github.com/apache/ambari-logsearch/pull/66
This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/cloud_mode.md b/docs/cloud_mode.md index 731c6d2b85..f3fe5dd348 100644 --- a/docs/cloud_mode.md +++ b/docs/cloud_mode.md @@ -15,6 +15,26 @@ See the License for the specific language governing permissions and limitations under the License. --> -### Log Feeder: cloud mode +### Log Feeder: Cloud Mode -TODO \ No newline at end of file +Log Feeder is responsible to ship logs to cloud storage. In Log Search case, it is a search engine (Solr), and that can be used by Log Search server to visualize logs. +Although there is a way to send logs to cloud storage / HDFS as well, of course in that case Solr is not used, so you won't be able to visualize the data. Log Feeder has 3 main modes: + +- `DEFAULT`: logs are shipped to Solr (or whatever is configured as output) +- `CLOUD`: logs are shipped to Cloud storage or HDFS (logs as text files, or in json format) +- `HYBRID`: logs are shipped to Solr and Cloud storage (or HDFS) as well (parallel mode) + +The cloud mode can be set by `logfeeder.cloud.storage.mode` property in `logfeeder.properties`. + +#### How it works ? + +In cloud mode, instead of shipping monitored logs to a specific location (directly) like into Solr, first it will ship the all monitored logs to new files to a temporal folder. Those logs are archived periodically, (you can check the configuration options about the log rollover in [logfeeder.properties](#logfeeder_properties.md) with `logfeeder.cloud.rollover.*` prefixes) and in every minutes, there is a background thread that will try to upload those archived logs to HDFS or cloud storage. + +HDFS client is responsible to do the upload and you can provide a `core-site.xml` on the classpath (the HDFS client can be configured to use different filesystems like s3a,wasb,gcs etc.), but if you do not have any `core-site.xml` on the classpath, you can provide `fs.*` properties in `logfeeder.properties` configuration. + +It is common, on specific clusters, there are both HDFS and cloud storage are used, so it is not valid to use the same filesystem that `core-site.xml` uses, that is why we need an option to override the filesystem valide (`fs.defaultFS`), that property is `logfeeder.cloud.storage.base.path` (e.g.: `s3a://mybucket/my/path`) + +Overall - as a minimum - it is enough to provide this 2 properties to enable cloud storage mode: + +- `logfeeder.cloud.storage.mode`: CLOUD +- `logfeeder.cloud.storage.base.path`: s3a://mybucket/apps/logsearch ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
