Robert Nettleton created AMBARI-20819:
-----------------------------------------
Summary: LogSearch Integration should limit requests to portal for
missing components
Key: AMBARI-20819
URL: https://issues.apache.org/jira/browse/AMBARI-20819
Project: Ambari
Issue Type: Bug
Components: ambari-server
Affects Versions: 2.5.0
Reporter: Robert Nettleton
Assignee: Robert Nettleton
Priority: Critical
Fix For: 2.5.1
The LogSearch Integration layer in Ambari typically caches logging metadata for
each host/component combination in a given cluster. This cache is configurable,
and is typically a long-lived cache, since the logging metadata is generally
not likely to change.
This issue was found during a test of LogSearch at a large scale, with a
cluster of over 800 nodes.
it was discovered that the LogSearch Integration layer continues to make
repeated requests for logging metadata for components that are not properly
configured in LogSearch.
The LogSearch portal logs show the following repeated requests on the "Perf"
cluster:
{code}
{"level":"INFO","file":"SolrDaoBase.java","thread_name":"qtp1018937824-280","line_number":61,"logger_name":"org.apache.ambari.logsearch.dao.SolrDaoBase","logtime":"1487951210778","log_message":"Solr
query will be processed:
q=*:*&start=0&rows=1&fq=host:perf\\-d\\-86.c.pramod\\-thangali.internal&fq=type:hst_agent&fq=logtime:[*+TO+*]&sort=logtime+desc,seq_num+desc"}
{"level":"INFO","file":"SolrDaoBase.java","thread_name":"qtp1018937824-273","line_number":61,"logger_name":"org.apache.ambari.logsearch.dao.SolrDaoBase","logtime":"1487951210797","log_message":"Solr
query will be processed:
q=*:*&start=0&rows=1&fq=host:perf\\-k\\-24.c.pramod\\-thangali.internal&fq=type:hst_agent&fq=logtime:[*+TO+*]&sort=logtime+desc,seq_num+desc"}
{"level":"INFO","file":"SolrDaoBase.java","thread_name":"qtp1018937824-275","line_number":61,"logger_name":"org.apache.ambari.logsearch.dao.SolrDaoBase","logtime":"1487951210810","log_message":"Solr
query will be processed:
q=*:*&start=0&rows=1&fq=host:perf\\-d\\-88.c.pramod\\-thangali.internal&fq=type:hst_agent&fq=logtime:[*+TO+*]&sort=logtime+desc,seq_num+desc"}
{code}
The SmartSense component, managed outside of Ambari, includes logging component
IDs the SmartSense stack definition, but does not include the proper metadata
to allow LogSearch Portal to properly handle the SmartSense logs.
In the current implementation of the integration layer, since the SmartSense
components are never found, the integration will repeatedly check the cache,
fail that check, and attempt a remote call to obtain the metadata. This in
turn can cause LogSearch to execute repeated, and un-necessary, queries against
the Solr database, which could potentially be a problem in larger deployments.
The LogSearch Integration in Ambari should be modified to have a maximum number
of failed attempts to tolerate. Having passed a maximum number of failed
attempts to obtain the logging metadata for a given component, the integration
code should refrain from further attempts to contact the LogSearch portal for
this component, and log a warning to indicate that the requested component is
not properly handled by LogSearch.
This will probably need to be fixed in:
{code}org.apache.ambari.server.controller.logging.LogSearchDataRetrievalService{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
