jscheffl commented on code in PR #56640:
URL: https://github.com/apache/airflow/pull/56640#discussion_r2433818870
##########
airflow-core/docs/security/audit_logs.rst:
##########
@@ -18,50 +18,607 @@
Audit Logs in Airflow
=====================
+Understanding Audit Logs
+-------------------------
-Overview
----------
+Audit logs serve as the historical record of an Airflow system, documenting
who performed what actions and when they occurred. These logs are essential for
maintaining system integrity, meeting compliance requirements, and conducting
forensic analysis when issues arise.
-Audit logs are a critical component of any system that needs to maintain a
high level of security and compliance.
-They provide a way to track user actions and system events, which can be used
to troubleshoot issues, detect security breaches, and ensure regulatory
compliance.
+In essence, audit logs answer three fundamental questions:
-In Airflow, audit logs are used to track user actions and system events that
occur during the execution of Dags and tasks.
-They are stored in a database and can be accessed through the Airflow UI.
+- **Who**: Which user or system component initiated an action
+- **What**: The specific operation that was performed
+- **When**: The precise timestamp of the event
-To be able to see audit logs, a user needs to have the ``Audit Logs.can_read``
permission. Such user will be able to see all audit logs, independently of the
Dags permissions applied.
+The primary purposes of audit logs include:
+- **Regulatory Compliance**: Meeting requirements for data governance and
audit trails
+- **Security Monitoring**: Detecting unauthorized access or suspicious
activities
+- **Operational Troubleshooting**: Understanding the sequence of events
leading to system issues
+- **Change Management**: Tracking modifications to critical system components
-Level of Audit Logs
+.. note::
+ Access to audit logs requires the ``Audit Logs.can_read`` permission. Users
with this permission can view all audit entries regardless of their
DAG-specific access rights.
+
+
+Understanding Event Logs
+-------------------------
+
+Event logs represent the operational heartbeat of an Airflow system. Unlike
audit logs, which focus on accountability and compliance, event logs capture
the technical details of system behavior, application performance, and
operational metrics.
+
+Event logs serve several critical functions:
+
+- **Debugging and Troubleshooting**: Providing detailed error messages and
stack traces
+- **Performance Monitoring**: Recording execution times, resource usage, and
system metrics
+- **Operational Insights**: Tracking system health, component interactions,
and workflow execution
+- **Development Support**: Offering detailed information for code debugging
and optimization
+
+Event logs are typically stored in log files or external logging systems and
include information such as:
+
+- Task execution details and output
+- System errors and warnings
+- Performance metrics and timing information
+- Component startup and shutdown events
+- Resource utilization data
+
+Audit Logs vs Event Logs
+------------------------------------------
+
+While both logging systems are crucial for system management, they serve
distinct purposes and audiences:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 25 37 38
+
+ * - Characteristic
+ - Audit Logs
+ - Event Logs
+ * - **Primary Purpose**
+ - Accountability and compliance tracking
+ - Operational monitoring and system debugging
+ * - **Target Audience**
+ - Security teams, auditors, compliance officers
+ - Developers, system administrators, operations teams
+ * - **Content Focus**
+ - User actions and administrative changes
+ - System behavior, errors, and performance data
+ * - **Storage Location**
+ - Structured database table (``log``)
+ - Log files, external logging systems
+ * - **Retention Requirements**
+ - Long-term (months to years for compliance)
+ - Short to medium-term (days to weeks)
+ * - **Query Patterns**
+ - "Who modified this configuration?"
+ - "Why did this task execution fail?"
Review Comment:
I am not fully with this, might be better as:
```suggestion
- No query made except is a log aggregation framework is used. Usually
logs are read on a per task execution basis and will describe: "Why did this
task execution fail?"
```
##########
airflow-core/docs/security/audit_logs.rst:
##########
@@ -18,50 +18,607 @@
Audit Logs in Airflow
=====================
+Understanding Audit Logs
+-------------------------
-Overview
----------
+Audit logs serve as the historical record of an Airflow system, documenting
who performed what actions and when they occurred. These logs are essential for
maintaining system integrity, meeting compliance requirements, and conducting
forensic analysis when issues arise.
-Audit logs are a critical component of any system that needs to maintain a
high level of security and compliance.
-They provide a way to track user actions and system events, which can be used
to troubleshoot issues, detect security breaches, and ensure regulatory
compliance.
+In essence, audit logs answer three fundamental questions:
-In Airflow, audit logs are used to track user actions and system events that
occur during the execution of Dags and tasks.
-They are stored in a database and can be accessed through the Airflow UI.
+- **Who**: Which user or system component initiated an action
+- **What**: The specific operation that was performed
+- **When**: The precise timestamp of the event
-To be able to see audit logs, a user needs to have the ``Audit Logs.can_read``
permission. Such user will be able to see all audit logs, independently of the
Dags permissions applied.
+The primary purposes of audit logs include:
+- **Regulatory Compliance**: Meeting requirements for data governance and
audit trails
+- **Security Monitoring**: Detecting unauthorized access or suspicious
activities
+- **Operational Troubleshooting**: Understanding the sequence of events
leading to system issues
+- **Change Management**: Tracking modifications to critical system components
-Level of Audit Logs
+.. note::
+ Access to audit logs requires the ``Audit Logs.can_read`` permission. Users
with this permission can view all audit entries regardless of their
DAG-specific access rights.
+
+
+Understanding Event Logs
+-------------------------
+
+Event logs represent the operational heartbeat of an Airflow system. Unlike
audit logs, which focus on accountability and compliance, event logs capture
the technical details of system behavior, application performance, and
operational metrics.
+
+Event logs serve several critical functions:
+
+- **Debugging and Troubleshooting**: Providing detailed error messages and
stack traces
+- **Performance Monitoring**: Recording execution times, resource usage, and
system metrics
+- **Operational Insights**: Tracking system health, component interactions,
and workflow execution
+- **Development Support**: Offering detailed information for code debugging
and optimization
+
+Event logs are typically stored in log files or external logging systems and
include information such as:
+
+- Task execution details and output
+- System errors and warnings
+- Performance metrics and timing information
+- Component startup and shutdown events
+- Resource utilization data
+
+Audit Logs vs Event Logs
+------------------------------------------
Review Comment:
Same as above
```suggestion
------------------------
```
##########
airflow-core/docs/security/audit_logs.rst:
##########
@@ -18,50 +18,607 @@
Audit Logs in Airflow
=====================
+Understanding Audit Logs
+-------------------------
-Overview
----------
+Audit logs serve as the historical record of an Airflow system, documenting
who performed what actions and when they occurred. These logs are essential for
maintaining system integrity, meeting compliance requirements, and conducting
forensic analysis when issues arise.
-Audit logs are a critical component of any system that needs to maintain a
high level of security and compliance.
-They provide a way to track user actions and system events, which can be used
to troubleshoot issues, detect security breaches, and ensure regulatory
compliance.
+In essence, audit logs answer three fundamental questions:
-In Airflow, audit logs are used to track user actions and system events that
occur during the execution of Dags and tasks.
-They are stored in a database and can be accessed through the Airflow UI.
+- **Who**: Which user or system component initiated an action
+- **What**: The specific operation that was performed
+- **When**: The precise timestamp of the event
-To be able to see audit logs, a user needs to have the ``Audit Logs.can_read``
permission. Such user will be able to see all audit logs, independently of the
Dags permissions applied.
+The primary purposes of audit logs include:
+- **Regulatory Compliance**: Meeting requirements for data governance and
audit trails
+- **Security Monitoring**: Detecting unauthorized access or suspicious
activities
+- **Operational Troubleshooting**: Understanding the sequence of events
leading to system issues
+- **Change Management**: Tracking modifications to critical system components
-Level of Audit Logs
+.. note::
+ Access to audit logs requires the ``Audit Logs.can_read`` permission. Users
with this permission can view all audit entries regardless of their
DAG-specific access rights.
+
+
+Understanding Event Logs
+-------------------------
Review Comment:
Same as above
```suggestion
------------------------
```
##########
airflow-core/docs/security/audit_logs.rst:
##########
@@ -18,50 +18,607 @@
Audit Logs in Airflow
=====================
+Understanding Audit Logs
+-------------------------
-Overview
----------
+Audit logs serve as the historical record of an Airflow system, documenting
who performed what actions and when they occurred. These logs are essential for
maintaining system integrity, meeting compliance requirements, and conducting
forensic analysis when issues arise.
-Audit logs are a critical component of any system that needs to maintain a
high level of security and compliance.
-They provide a way to track user actions and system events, which can be used
to troubleshoot issues, detect security breaches, and ensure regulatory
compliance.
+In essence, audit logs answer three fundamental questions:
-In Airflow, audit logs are used to track user actions and system events that
occur during the execution of Dags and tasks.
-They are stored in a database and can be accessed through the Airflow UI.
+- **Who**: Which user or system component initiated an action
+- **What**: The specific operation that was performed
+- **When**: The precise timestamp of the event
-To be able to see audit logs, a user needs to have the ``Audit Logs.can_read``
permission. Such user will be able to see all audit logs, independently of the
Dags permissions applied.
+The primary purposes of audit logs include:
+- **Regulatory Compliance**: Meeting requirements for data governance and
audit trails
+- **Security Monitoring**: Detecting unauthorized access or suspicious
activities
+- **Operational Troubleshooting**: Understanding the sequence of events
leading to system issues
+- **Change Management**: Tracking modifications to critical system components
-Level of Audit Logs
+.. note::
+ Access to audit logs requires the ``Audit Logs.can_read`` permission. Users
with this permission can view all audit entries regardless of their
DAG-specific access rights.
+
+
+Understanding Event Logs
+-------------------------
+
+Event logs represent the operational heartbeat of an Airflow system. Unlike
audit logs, which focus on accountability and compliance, event logs capture
the technical details of system behavior, application performance, and
operational metrics.
+
+Event logs serve several critical functions:
+
+- **Debugging and Troubleshooting**: Providing detailed error messages and
stack traces
+- **Performance Monitoring**: Recording execution times, resource usage, and
system metrics
+- **Operational Insights**: Tracking system health, component interactions,
and workflow execution
+- **Development Support**: Offering detailed information for code debugging
and optimization
+
+Event logs are typically stored in log files or external logging systems and
include information such as:
+
+- Task execution details and output
+- System errors and warnings
+- Performance metrics and timing information
+- Component startup and shutdown events
+- Resource utilization data
+
+Audit Logs vs Event Logs
+------------------------------------------
+
+While both logging systems are crucial for system management, they serve
distinct purposes and audiences:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 25 37 38
+
+ * - Characteristic
+ - Audit Logs
+ - Event Logs
+ * - **Primary Purpose**
+ - Accountability and compliance tracking
+ - Operational monitoring and system debugging
+ * - **Target Audience**
+ - Security teams, auditors, compliance officers
+ - Developers, system administrators, operations teams
+ * - **Content Focus**
+ - User actions and administrative changes
+ - System behavior, errors, and performance data
+ * - **Storage Location**
+ - Structured database table (``log``)
+ - Log files, external logging systems
+ * - **Retention Requirements**
+ - Long-term (months to years for compliance)
Review Comment:
```suggestion
- Long-term (months to years for compliance), if not purged from
database
```
##########
airflow-core/docs/security/audit_logs.rst:
##########
@@ -18,50 +18,607 @@
Audit Logs in Airflow
=====================
+Understanding Audit Logs
+-------------------------
Review Comment:
Nit: underline chars should match headline length
```suggestion
------------------------
```
##########
airflow-core/docs/security/audit_logs.rst:
##########
@@ -18,50 +18,607 @@
Audit Logs in Airflow
=====================
+Understanding Audit Logs
+-------------------------
-Overview
----------
+Audit logs serve as the historical record of an Airflow system, documenting
who performed what actions and when they occurred. These logs are essential for
maintaining system integrity, meeting compliance requirements, and conducting
forensic analysis when issues arise.
-Audit logs are a critical component of any system that needs to maintain a
high level of security and compliance.
-They provide a way to track user actions and system events, which can be used
to troubleshoot issues, detect security breaches, and ensure regulatory
compliance.
+In essence, audit logs answer three fundamental questions:
-In Airflow, audit logs are used to track user actions and system events that
occur during the execution of Dags and tasks.
-They are stored in a database and can be accessed through the Airflow UI.
+- **Who**: Which user or system component initiated an action
+- **What**: The specific operation that was performed
+- **When**: The precise timestamp of the event
-To be able to see audit logs, a user needs to have the ``Audit Logs.can_read``
permission. Such user will be able to see all audit logs, independently of the
Dags permissions applied.
+The primary purposes of audit logs include:
+- **Regulatory Compliance**: Meeting requirements for data governance and
audit trails
+- **Security Monitoring**: Detecting unauthorized access or suspicious
activities
+- **Operational Troubleshooting**: Understanding the sequence of events
leading to system issues
+- **Change Management**: Tracking modifications to critical system components
-Level of Audit Logs
+.. note::
+ Access to audit logs requires the ``Audit Logs.can_read`` permission. Users
with this permission can view all audit entries regardless of their
DAG-specific access rights.
+
+
+Understanding Event Logs
+-------------------------
+
+Event logs represent the operational heartbeat of an Airflow system. Unlike
audit logs, which focus on accountability and compliance, event logs capture
the technical details of system behavior, application performance, and
operational metrics.
+
+Event logs serve several critical functions:
+
+- **Debugging and Troubleshooting**: Providing detailed error messages and
stack traces
+- **Performance Monitoring**: Recording execution times, resource usage, and
system metrics
+- **Operational Insights**: Tracking system health, component interactions,
and workflow execution
+- **Development Support**: Offering detailed information for code debugging
and optimization
+
+Event logs are typically stored in log files or external logging systems and
include information such as:
+
+- Task execution details and output
+- System errors and warnings
+- Performance metrics and timing information
+- Component startup and shutdown events
+- Resource utilization data
+
+Audit Logs vs Event Logs
+------------------------------------------
+
+While both logging systems are crucial for system management, they serve
distinct purposes and audiences:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 25 37 38
+
+ * - Characteristic
+ - Audit Logs
+ - Event Logs
+ * - **Primary Purpose**
+ - Accountability and compliance tracking
+ - Operational monitoring and system debugging
+ * - **Target Audience**
+ - Security teams, auditors, compliance officers
+ - Developers, system administrators, operations teams
+ * - **Content Focus**
+ - User actions and administrative changes
+ - System behavior, errors, and performance data
+ * - **Storage Location**
+ - Structured database table (``log``)
+ - Log files, external logging systems
+ * - **Retention Requirements**
+ - Long-term (months to years for compliance)
+ - Short to medium-term (days to weeks)
+ * - **Query Patterns**
+ - "Who modified this configuration?"
+ - "Why did this task execution fail?"
+
+
+Accessing Audit Logs
--------------------
-Audit logs exist at the task level and the user level.
+Airflow provides multiple interfaces for accessing audit log data, each suited
to different use cases and technical requirements:
-- Task Level: At the task level, audit logs capture information related to the
execution of a task, such as the start time, end time, and status of the task.
+**Web User Interface**
+ The Airflow web interface provides the most accessible method for viewing
audit logs. Navigate to **Browse → Audit Logs** to access an interface with
built-in filtering, sorting, and search capabilities. This interface is ideal
for ad-hoc investigations and routine monitoring.
-- User Level: At the user level, audit logs capture information related to
user actions, such as creating, modifying, or deleting a Dag or task.
+**REST API Integration**
+ For programmatic access and system integration, use the ``/eventLogs`` REST
API endpoint. This approach enables automated monitoring, integration with
external security tools, and custom reporting applications.
+**Direct Database Access**
+ Advanced users can query the ``log`` table directly using SQL. This method
provides maximum flexibility for complex queries, custom reporting, and
integration with business intelligence tools.
Review Comment:
We should not recommend quering the database directly. This is not a public
API. Everybody can make this at own risk only.
##########
airflow-core/docs/security/audit_logs.rst:
##########
@@ -18,50 +18,607 @@
Audit Logs in Airflow
=====================
+Understanding Audit Logs
+-------------------------
-Overview
----------
+Audit logs serve as the historical record of an Airflow system, documenting
who performed what actions and when they occurred. These logs are essential for
maintaining system integrity, meeting compliance requirements, and conducting
forensic analysis when issues arise.
-Audit logs are a critical component of any system that needs to maintain a
high level of security and compliance.
-They provide a way to track user actions and system events, which can be used
to troubleshoot issues, detect security breaches, and ensure regulatory
compliance.
+In essence, audit logs answer three fundamental questions:
-In Airflow, audit logs are used to track user actions and system events that
occur during the execution of Dags and tasks.
-They are stored in a database and can be accessed through the Airflow UI.
+- **Who**: Which user or system component initiated an action
+- **What**: The specific operation that was performed
+- **When**: The precise timestamp of the event
-To be able to see audit logs, a user needs to have the ``Audit Logs.can_read``
permission. Such user will be able to see all audit logs, independently of the
Dags permissions applied.
+The primary purposes of audit logs include:
+- **Regulatory Compliance**: Meeting requirements for data governance and
audit trails
+- **Security Monitoring**: Detecting unauthorized access or suspicious
activities
+- **Operational Troubleshooting**: Understanding the sequence of events
leading to system issues
+- **Change Management**: Tracking modifications to critical system components
-Level of Audit Logs
+.. note::
+ Access to audit logs requires the ``Audit Logs.can_read`` permission. Users
with this permission can view all audit entries regardless of their
DAG-specific access rights.
+
+
+Understanding Event Logs
+-------------------------
+
+Event logs represent the operational heartbeat of an Airflow system. Unlike
audit logs, which focus on accountability and compliance, event logs capture
the technical details of system behavior, application performance, and
operational metrics.
+
+Event logs serve several critical functions:
+
+- **Debugging and Troubleshooting**: Providing detailed error messages and
stack traces
+- **Performance Monitoring**: Recording execution times, resource usage, and
system metrics
+- **Operational Insights**: Tracking system health, component interactions,
and workflow execution
+- **Development Support**: Offering detailed information for code debugging
and optimization
+
+Event logs are typically stored in log files or external logging systems and
include information such as:
+
+- Task execution details and output
+- System errors and warnings
+- Performance metrics and timing information
+- Component startup and shutdown events
+- Resource utilization data
+
+Audit Logs vs Event Logs
+------------------------------------------
+
+While both logging systems are crucial for system management, they serve
distinct purposes and audiences:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 25 37 38
+
+ * - Characteristic
+ - Audit Logs
+ - Event Logs
+ * - **Primary Purpose**
+ - Accountability and compliance tracking
+ - Operational monitoring and system debugging
+ * - **Target Audience**
+ - Security teams, auditors, compliance officers
+ - Developers, system administrators, operations teams
+ * - **Content Focus**
+ - User actions and administrative changes
+ - System behavior, errors, and performance data
+ * - **Storage Location**
+ - Structured database table (``log``)
+ - Log files, external logging systems
+ * - **Retention Requirements**
+ - Long-term (months to years for compliance)
+ - Short to medium-term (days to weeks)
+ * - **Query Patterns**
+ - "Who modified this configuration?"
Review Comment:
This is not full right, in case of Airflow no configuration is changed.
```suggestion
- "Who cleared the task instance for re-execution?"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
