Pavan Badempet created SPARK-59660:
--------------------------------------
Summary: [PySpark][SQL] Native DataFrame lineage, pre-flight
linting, and DAG visualization (df.lineage)
Key: SPARK-59660
URL: https://issues.apache.org/jira/browse/SPARK-59660
Project: Spark
Issue Type: New Feature
Components: PySpark, SQL
Affects Versions: 4.2.0
Reporter: Pavan Badempet
h2. Summary
Introduce a native .lineage accessor on pyspark.sql.DataFrame to provide
zero-overhead field-level (column) lineage tracing, pre-flight Catalyst
execution plan linting, and interactive in-notebook DAG visualization.
h2. Motivation & Problem Statement
1. *Observability Gap:* Currently, df.explain() emits raw nested ASCII trees
that are difficult to inspect programmatically. External lineage agents (e.g.
OpenLineage, Spline) require heavy JVM bytecode instrumentation, network
daemons, and external database servers (e.g. Marquez).
2. *Pre-Flight Cost Prevention:* In cloud environments (Databricks, AWS EMR,
GCP Dataproc), developers frequently submit expensive Spark actions (.count(),
.write()) with un-optimized plans (e.g., accidental Cartesian joins, unpruned
schema scans). Catching these anti-patterns before task scheduling prevents
cloud compute waste.
h2. Proposed User-Facing API
{code:python}
# 1. Pre-flight compiler checks before submitting actions
report = df.lineage.lint()
if report.has_critical_issues():
for issue in report.issues:
print(f"[\{issue.severity}] \{issue.code}: \{issue.message}")
# 2. Field-to-field column lineage
col_lineage = df.lineage.columns
# Returns: \{'total_amount': ColumnLineage(sources=['transactions.amount',
'rates.multiplier'])}
# 3. Interactive DAG visualization in Databricks / Jupyter notebooks
df.lineage.display()
# 4. Export to documentation formats
mermaid_diagram = df.lineage.to_mermaid()
html_doc = df.lineage.to_html()
{code}
h2. Key Characteristics
- Zero external runtime dependencies (client-side Catalyst AST parsing).
- Enterprise credential sanitization (automatic redaction of AWS/GCP tokens and
JDBC passwords).
- Reference implementation tested against PySpark 3.2 through 4.2.0.
h2. Related Precedents
Similar to Koalas (SPARK-34849 -> pyspark.pandas) and spark-xml (SPARK-47000),
this capability has been designed and tested as a standalone module before
upstream integration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
