[
https://issues.apache.org/jira/browse/HIVE-29732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Turoczy updated HIVE-29732:
----------------------------------
Description:
Hive currently generates a large number of log files across its components,
including HiveServer2, Hive Metastore, background services, query execution,
and related operational processes.
Although these logs contain valuable diagnostic and operational information,
consuming them typically requires platform-specific log collection
configuration. This makes troubleshooting, monitoring, and cloud deployment
more difficult and often results in custom solutions for each environment.
We should investigate and implement a modular, cloud-native log export
framework that allows Hive logs to be consumed by external observability
platforms such as:
* Datadog
* Azure Monitor / Log Analytics
* Other OpenTelemetry-compatible or standard log collection systems
The goal is not to introduce a hard dependency on a specific vendor. Hive
should expose logs through standard interfaces and structured formats, while
the deployment platform or operator decides where those logs are delivered.
h2. Goals
* Make Hive logs easier to collect, search, correlate, and analyze.
* Reduce the need for custom log parsing and environment-specific integrations.
* Support cloud-native deployments, including Kubernetes-based environments.
* Keep the implementation vendor-neutral and modular.
* Preserve compatibility with existing file-based logging.
* Allow integrations to be enabled or disabled through configuration.
* Improve correlation between Hive requests, queries, sessions, and execution
components.
h3. 1. Structured Logging
Introduce or improve support for structured log output, preferably JSON, in
addition to the existing text-based log format.
Structured log records should include relevant fields where available, such as:
* Timestamp
* Log level
* Component name
* Host or pod identifier
* Thread name
* Hive session ID
* Query ID
* Operation ID
* User
* Database or catalog
* Application name
* Error code
* Exception type
* Trace ID and span ID
* Execution engine
* Duration or elapsed time
Sensitive values, SQL text, credentials, tokens, and configuration secrets must
be handled according to existing Hive security and redaction policies.
h3. 2. Standard Output Support
Hive services should support writing structured logs to standard output and
standard error.
This should allow Kubernetes and cloud-native logging agents to collect logs
without requiring direct access to Hive log files.
File-based logging should remain available for backward compatibility.
h3. 3. OpenTelemetry-Compatible Integration (Separate task)
Investigate adding optional OpenTelemetry logging and tracing integration.
The preferred model should be:
Hive
→ structured logs / OpenTelemetry
→ collector or platform logging agent
→ Datadog, Azure Log Analytics, or another backend
Hive should not communicate directly with a specific vendor API unless
implemented as a separate optional module.
An OpenTelemetry Collector, Datadog Agent, Azure Monitor Agent, Fluent Bit, or
similar component should be responsible for forwarding and transforming the
logs.
h3. 4. Pluggable Log Export Interface
Introduce a modular log-export abstraction that allows additional exporters to
be implemented without changing Hive core components.
Potential modules could include:
{{hive-observability-core}}
{{hive-observability-otel}}
{{hive-observability-json}}
Vendor-specific integrations, if required, should remain external or optional:
{{hive-observability-datadog}}
{{hive-observability-azure-monitor}}
The core Hive distribution should not require Datadog, Azure, or other vendor
libraries by default.
h3. 5. Query and Request Correlation
Ensure that the same correlation identifiers are propagated across relevant
components, where technically possible.
For example:
{{Client request
→ HiveServer2 operation
→ query compilation
→ execution engine
→ Hive Metastore request
→ audit and operational logs}}
This would allow an administrator to search for a query ID or trace ID and
retrieve the related events across multiple Hive components.
h3. 6. Configuration
The integration should be configuration-driven.
Example configuration areas:
{{hive.log.format=json}}
{{hive.log.output=file,stdout}}
{{hive.observability.enabled=true}}
{{hive.observability.exporter=otel}}
{{hive.observability.service.name=hiveserver2}}
{{hive.observability.include.query.text=false}}
{{hive.observability.include.user=true}}
{{hive.observability.correlation.enabled=true}}
The exact property names should follow existing Hive configuration conventions.
h2. Datadog Integration Example (separate task)
A Datadog deployment should be able to collect Hive logs through the Datadog
Agent from:
* Container standard output
* Existing Hive log files
* An OpenTelemetry Collector
* An OpenTelemetry-compatible endpoint
Hive-specific attributes should be mapped to Datadog fields to support
filtering by query ID, session ID, user, component, cluster, and execution
engine.
No Datadog-specific dependency should be required in Hive core.
h2. Azure Integration Example (separate task)
An Azure deployment should be able to collect Hive logs using:
* Azure Monitor Agent
* Container Insights
* Azure Log Analytics
* An OpenTelemetry Collector configured with an Azure-compatible exporter
Hive should emit structured logs and correlation metadata that can be indexed
and queried in Log Analytics.
No Azure-specific dependency should be required in Hive core.
was:
Hive currently generates a large number of log files across its components,
including HiveServer2, Hive Metastore, background services, query execution,
and related operational processes.
Although these logs contain valuable diagnostic and operational information,
consuming them typically requires platform-specific log collection
configuration. This makes troubleshooting, monitoring, and cloud deployment
more difficult and often results in custom solutions for each environment.
We should investigate and implement a modular, cloud-native log export
framework that allows Hive logs to be consumed by external observability
platforms such as:
* Datadog
* Azure Monitor / Log Analytics
* Other OpenTelemetry-compatible or standard log collection systems
The goal is not to introduce a hard dependency on a specific vendor. Hive
should expose logs through standard interfaces and structured formats, while
the deployment platform or operator decides where those logs are delivered.
h2. Goals
* Make Hive logs easier to collect, search, correlate, and analyze.
* Reduce the need for custom log parsing and environment-specific integrations.
* Support cloud-native deployments, including Kubernetes-based environments.
* Keep the implementation vendor-neutral and modular.
* Preserve compatibility with existing file-based logging.
* Allow integrations to be enabled or disabled through configuration.
* Improve correlation between Hive requests, queries, sessions, and execution
components.
h3. 1. Structured Logging
Introduce or improve support for structured log output, preferably JSON, in
addition to the existing text-based log format.
Structured log records should include relevant fields where available, such as:
* Timestamp
* Log level
* Component name
* Host or pod identifier
* Thread name
* Hive session ID
* Query ID
* Operation ID
* User
* Database or catalog
* Application name
* Error code
* Exception type
* Trace ID and span ID
* Execution engine
* Duration or elapsed time
Sensitive values, SQL text, credentials, tokens, and configuration secrets must
be handled according to existing Hive security and redaction policies.
h3. 2. Standard Output Support
Hive services should support writing structured logs to standard output and
standard error.
This should allow Kubernetes and cloud-native logging agents to collect logs
without requiring direct access to Hive log files.
File-based logging should remain available for backward compatibility.
h3. 3. OpenTelemetry-Compatible Integration (Separate task)
Investigate adding optional OpenTelemetry logging and tracing integration.
The preferred model should be:
{{Hive
→ structured logs / OpenTelemetry
→ collector or platform logging agent
→ Datadog, Azure Log Analytics, or another backend}}
Hive should not communicate directly with a specific vendor API unless
implemented as a separate optional module.
An OpenTelemetry Collector, Datadog Agent, Azure Monitor Agent, Fluent Bit, or
similar component should be responsible for forwarding and transforming the
logs.
h3. 4. Pluggable Log Export Interface
Introduce a modular log-export abstraction that allows additional exporters to
be implemented without changing Hive core components.
Potential modules could include:
{{hive-observability-core
hive-observability-otel
hive-observability-json}}
Vendor-specific integrations, if required, should remain external or optional:
{{hive-observability-datadog
hive-observability-azure-monitor}}
The core Hive distribution should not require Datadog, Azure, or other vendor
libraries by default.
h3. 5. Query and Request Correlation
Ensure that the same correlation identifiers are propagated across relevant
components, where technically possible.
For example:
{{Client request
→ HiveServer2 operation
→ query compilation
→ execution engine
→ Hive Metastore request
→ audit and operational logs}}
This would allow an administrator to search for a query ID or trace ID and
retrieve the related events across multiple Hive components.
h3. 6. Configuration
The integration should be configuration-driven.
Example configuration areas:
{{hive.log.format=json
hive.log.output=file,stdout
hive.observability.enabled=true
hive.observability.exporter=otel
hive.observability.service.name=hiveserver2
hive.observability.include.query.text=false
hive.observability.include.user=true
hive.observability.correlation.enabled=true}}
The exact property names should follow existing Hive configuration conventions.
h2. Datadog Integration Example (separate task)
A Datadog deployment should be able to collect Hive logs through the Datadog
Agent from:
* Container standard output
* Existing Hive log files
* An OpenTelemetry Collector
* An OpenTelemetry-compatible endpoint
Hive-specific attributes should be mapped to Datadog fields to support
filtering by query ID, session ID, user, component, cluster, and execution
engine.
No Datadog-specific dependency should be required in Hive core.
h2. Azure Integration Example (separate task)
An Azure deployment should be able to collect Hive logs using:
* Azure Monitor Agent
* Container Insights
* Azure Log Analytics
* An OpenTelemetry Collector configured with an Azure-compatible exporter
Hive should emit structured logs and correlation metadata that can be indexed
and queried in Log Analytics.
No Azure-specific dependency should be required in Hive core.
> Introduce a Modular Log Export and Observability Integration Framework for
> Hive
> -------------------------------------------------------------------------------
>
> Key: HIVE-29732
> URL: https://issues.apache.org/jira/browse/HIVE-29732
> Project: Hive
> Issue Type: New Feature
> Reporter: Attila Turoczy
> Priority: Major
> Labels: cloud
>
> Hive currently generates a large number of log files across its components,
> including HiveServer2, Hive Metastore, background services, query execution,
> and related operational processes.
> Although these logs contain valuable diagnostic and operational information,
> consuming them typically requires platform-specific log collection
> configuration. This makes troubleshooting, monitoring, and cloud deployment
> more difficult and often results in custom solutions for each environment.
> We should investigate and implement a modular, cloud-native log export
> framework that allows Hive logs to be consumed by external observability
> platforms such as:
> * Datadog
> * Azure Monitor / Log Analytics
> * Other OpenTelemetry-compatible or standard log collection systems
> The goal is not to introduce a hard dependency on a specific vendor. Hive
> should expose logs through standard interfaces and structured formats, while
> the deployment platform or operator decides where those logs are delivered.
> h2. Goals
> * Make Hive logs easier to collect, search, correlate, and analyze.
> * Reduce the need for custom log parsing and environment-specific
> integrations.
> * Support cloud-native deployments, including Kubernetes-based environments.
> * Keep the implementation vendor-neutral and modular.
> * Preserve compatibility with existing file-based logging.
> * Allow integrations to be enabled or disabled through configuration.
> * Improve correlation between Hive requests, queries, sessions, and
> execution components.
> h3. 1. Structured Logging
> Introduce or improve support for structured log output, preferably JSON, in
> addition to the existing text-based log format.
> Structured log records should include relevant fields where available, such
> as:
> * Timestamp
> * Log level
> * Component name
> * Host or pod identifier
> * Thread name
> * Hive session ID
> * Query ID
> * Operation ID
> * User
> * Database or catalog
> * Application name
> * Error code
> * Exception type
> * Trace ID and span ID
> * Execution engine
> * Duration or elapsed time
> Sensitive values, SQL text, credentials, tokens, and configuration secrets
> must be handled according to existing Hive security and redaction policies.
> h3. 2. Standard Output Support
> Hive services should support writing structured logs to standard output and
> standard error.
> This should allow Kubernetes and cloud-native logging agents to collect logs
> without requiring direct access to Hive log files.
> File-based logging should remain available for backward compatibility.
> h3. 3. OpenTelemetry-Compatible Integration (Separate task)
> Investigate adding optional OpenTelemetry logging and tracing integration.
> The preferred model should be:
> Hive
> → structured logs / OpenTelemetry
> → collector or platform logging agent
> → Datadog, Azure Log Analytics, or another backend
> Hive should not communicate directly with a specific vendor API unless
> implemented as a separate optional module.
> An OpenTelemetry Collector, Datadog Agent, Azure Monitor Agent, Fluent Bit,
> or similar component should be responsible for forwarding and transforming
> the logs.
> h3. 4. Pluggable Log Export Interface
> Introduce a modular log-export abstraction that allows additional exporters
> to be implemented without changing Hive core components.
> Potential modules could include:
> {{hive-observability-core}}
> {{hive-observability-otel}}
> {{hive-observability-json}}
> Vendor-specific integrations, if required, should remain external or optional:
> {{hive-observability-datadog}}
> {{hive-observability-azure-monitor}}
> The core Hive distribution should not require Datadog, Azure, or other vendor
> libraries by default.
> h3. 5. Query and Request Correlation
> Ensure that the same correlation identifiers are propagated across relevant
> components, where technically possible.
> For example:
> {{Client request
> → HiveServer2 operation
> → query compilation
> → execution engine
> → Hive Metastore request
> → audit and operational logs}}
> This would allow an administrator to search for a query ID or trace ID and
> retrieve the related events across multiple Hive components.
> h3. 6. Configuration
> The integration should be configuration-driven.
> Example configuration areas:
> {{hive.log.format=json}}
> {{hive.log.output=file,stdout}}
> {{hive.observability.enabled=true}}
> {{hive.observability.exporter=otel}}
> {{hive.observability.service.name=hiveserver2}}
> {{hive.observability.include.query.text=false}}
> {{hive.observability.include.user=true}}
> {{hive.observability.correlation.enabled=true}}
> The exact property names should follow existing Hive configuration
> conventions.
> h2. Datadog Integration Example (separate task)
> A Datadog deployment should be able to collect Hive logs through the Datadog
> Agent from:
> * Container standard output
> * Existing Hive log files
> * An OpenTelemetry Collector
> * An OpenTelemetry-compatible endpoint
> Hive-specific attributes should be mapped to Datadog fields to support
> filtering by query ID, session ID, user, component, cluster, and execution
> engine.
> No Datadog-specific dependency should be required in Hive core.
> h2. Azure Integration Example (separate task)
> An Azure deployment should be able to collect Hive logs using:
> * Azure Monitor Agent
> * Container Insights
> * Azure Log Analytics
> * An OpenTelemetry Collector configured with an Azure-compatible exporter
> Hive should emit structured logs and correlation metadata that can be indexed
> and queried in Log Analytics.
> No Azure-specific dependency should be required in Hive core.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
