[
https://issues.apache.org/jira/browse/HIVE-29732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Turoczy updated HIVE-29732:
----------------------------------
Labels: cloud (was: )
> Introduce a Modular Log Export and Observability Integration Framework for
> Hive
> -------------------------------------------------------------------------------
>
> Key: HIVE-29732
> URL: https://issues.apache.org/jira/browse/HIVE-29732
> Project: Hive
> Issue Type: New Feature
> Reporter: Attila Turoczy
> Priority: Major
> Labels: cloud
>
> Hive currently generates a large number of log files across its components,
> including HiveServer2, Hive Metastore, background services, query execution,
> and related operational processes.
> Although these logs contain valuable diagnostic and operational information,
> consuming them typically requires platform-specific log collection
> configuration. This makes troubleshooting, monitoring, and cloud deployment
> more difficult and often results in custom solutions for each environment.
> We should investigate and implement a modular, cloud-native log export
> framework that allows Hive logs to be consumed by external observability
> platforms such as:
> * Datadog
> * Azure Monitor / Log Analytics
> * Other OpenTelemetry-compatible or standard log collection systems
> The goal is not to introduce a hard dependency on a specific vendor. Hive
> should expose logs through standard interfaces and structured formats, while
> the deployment platform or operator decides where those logs are delivered.
> h2. Goals
> * Make Hive logs easier to collect, search, correlate, and analyze.
> * Reduce the need for custom log parsing and environment-specific
> integrations.
> * Support cloud-native deployments, including Kubernetes-based environments.
> * Keep the implementation vendor-neutral and modular.
> * Preserve compatibility with existing file-based logging.
> * Allow integrations to be enabled or disabled through configuration.
> * Improve correlation between Hive requests, queries, sessions, and
> execution components.
> h3. 1. Structured Logging
> Introduce or improve support for structured log output, preferably JSON, in
> addition to the existing text-based log format.
> Structured log records should include relevant fields where available, such
> as:
> * Timestamp
> * Log level
> * Component name
> * Host or pod identifier
> * Thread name
> * Hive session ID
> * Query ID
> * Operation ID
> * User
> * Database or catalog
> * Application name
> * Error code
> * Exception type
> * Trace ID and span ID
> * Execution engine
> * Duration or elapsed time
> Sensitive values, SQL text, credentials, tokens, and configuration secrets
> must be handled according to existing Hive security and redaction policies.
> h3. 2. Standard Output Support
> Hive services should support writing structured logs to standard output and
> standard error.
> This should allow Kubernetes and cloud-native logging agents to collect logs
> without requiring direct access to Hive log files.
> File-based logging should remain available for backward compatibility.
> h3. 3. OpenTelemetry-Compatible Integration (Separate task)
> Investigate adding optional OpenTelemetry logging and tracing integration.
> The preferred model should be:
> {{Hive
> → structured logs / OpenTelemetry
> → collector or platform logging agent
> → Datadog, Azure Log Analytics, or another backend}}
> Hive should not communicate directly with a specific vendor API unless
> implemented as a separate optional module.
> An OpenTelemetry Collector, Datadog Agent, Azure Monitor Agent, Fluent Bit,
> or similar component should be responsible for forwarding and transforming
> the logs.
> h3. 4. Pluggable Log Export Interface
> Introduce a modular log-export abstraction that allows additional exporters
> to be implemented without changing Hive core components.
> Potential modules could include:
> {{hive-observability-core
> hive-observability-otel
> hive-observability-json}}
> Vendor-specific integrations, if required, should remain external or optional:
> {{hive-observability-datadog
> hive-observability-azure-monitor}}
> The core Hive distribution should not require Datadog, Azure, or other vendor
> libraries by default.
> h3. 5. Query and Request Correlation
> Ensure that the same correlation identifiers are propagated across relevant
> components, where technically possible.
> For example:
> {{Client request
> → HiveServer2 operation
> → query compilation
> → execution engine
> → Hive Metastore request
> → audit and operational logs}}
> This would allow an administrator to search for a query ID or trace ID and
> retrieve the related events across multiple Hive components.
> h3. 6. Configuration
> The integration should be configuration-driven.
> Example configuration areas:
> {{hive.log.format=json
> hive.log.output=file,stdout
> hive.observability.enabled=true
> hive.observability.exporter=otel
> hive.observability.service.name=hiveserver2
> hive.observability.include.query.text=false
> hive.observability.include.user=true
> hive.observability.correlation.enabled=true}}
> The exact property names should follow existing Hive configuration
> conventions.
> h2. Datadog Integration Example (separate task)
> A Datadog deployment should be able to collect Hive logs through the Datadog
> Agent from:
> * Container standard output
> * Existing Hive log files
> * An OpenTelemetry Collector
> * An OpenTelemetry-compatible endpoint
> Hive-specific attributes should be mapped to Datadog fields to support
> filtering by query ID, session ID, user, component, cluster, and execution
> engine.
> No Datadog-specific dependency should be required in Hive core.
> h2. Azure Integration Example (separate task)
> An Azure deployment should be able to collect Hive logs using:
> * Azure Monitor Agent
> * Container Insights
> * Azure Log Analytics
> * An OpenTelemetry Collector configured with an Azure-compatible exporter
> Hive should emit structured logs and correlation metadata that can be indexed
> and queried in Log Analytics.
> No Azure-specific dependency should be required in Hive core.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
