oleewere closed pull request #66: AMBARI-23456. Add cloud mode documentation
URL: https://github.com/apache/ambari-logsearch/pull/66

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docs/cloud_mode.md b/docs/cloud_mode.md
index 731c6d2b85..f3fe5dd348 100644
--- a/docs/cloud_mode.md
+++ b/docs/cloud_mode.md
@@ -15,6 +15,26 @@ See the License for the specific language governing 
permissions and
 limitations under the License.
-### Log Feeder: cloud mode
+### Log Feeder: Cloud Mode
\ No newline at end of file
+Log Feeder is responsible to ship logs to cloud storage. In Log Search case, 
it is a search engine (Solr), and that can be used by Log Search server to 
visualize logs.
+Although there is a way to send logs to cloud storage / HDFS as well, of 
course in that case Solr is not used, so you won't be able to visualize the 
data. Log Feeder has 3 main modes:
+- `DEFAULT`: logs are shipped to Solr (or whatever is configured as output)
+- `CLOUD`: logs are shipped to Cloud storage or HDFS (logs as text files, or 
in json format)
+- `HYBRID`: logs are shipped to Solr and Cloud storage (or HDFS) as well 
(parallel mode) 
+The cloud mode can be set by `logfeeder.cloud.storage.mode` property in 
+#### How it works ?
+In cloud mode, instead of shipping monitored logs to a specific location 
(directly) like into Solr, first it will ship the all monitored logs to new 
files to a temporal folder. Those logs are archived periodically, (you can 
check the configuration options about the log rollover in 
[logfeeder.properties](#logfeeder_properties.md) with 
`logfeeder.cloud.rollover.*` prefixes) and in every minutes, there is a 
background thread that will try to upload those archived logs to HDFS or cloud 
+HDFS client is responsible to do the upload and you can provide a 
`core-site.xml` on the classpath (the HDFS client can be configured to use 
different filesystems like s3a,wasb,gcs etc.), but if you do not have any 
`core-site.xml` on the classpath, you can provide `fs.*` properties in 
`logfeeder.properties` configuration. 
+It is common, on specific clusters, there are both HDFS and cloud storage are 
used, so it is not valid to use the same filesystem that `core-site.xml` uses, 
that is why we need an option to override the filesystem valide 
(`fs.defaultFS`), that property is `logfeeder.cloud.storage.base.path` (e.g.: 
+Overall - as a minimum - it is enough to provide this 2 properties to enable 
cloud storage mode:
+- `logfeeder.cloud.storage.mode`: CLOUD
+- `logfeeder.cloud.storage.base.path`: s3a://mybucket/apps/logsearch


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

Reply via email to