sumeetgajjar opened a new pull request, #5629: URL: https://github.com/apache/iceberg/pull/5629
SPARK-40067 added Table#name() to the BatchScan node on SparkUI. With this improvement in place, for all DSv2 tables, SparkUI shows `BatchScan <Table#name>` instead of uninformative `BatchScan`. In its current state, for iceberg table, SparkUI shows `BatchScan <catalog_name>.<namespace>.<table>`. This can be further improved by adding the fileFormat and tableFormatVersion to the table name thereby yielding the final result as `BatchScan iceberg/<fileFormat>/<tableFormatVersion> <catalog_name>.<namespace>.<table>`. E.g. `BatchScan iceberg/parquet/v1 spark_catalog.default.ui_test_parquet` Note: SPARK-40067 is available from Spark3.4 and onwards. Since it is not a bug fix, we cannot backport the improvement to OSS Spark3.3. So if one is using Iceberg with OSS Spark3.3 then the improvement won't be visible and uninformative `BatchScan` would still be shown on the SparkUI unless they backport it in their fork. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
