deniskuzZ commented on PR #6794:
URL: https://github.com/apache/hive/pull/6794#issuecomment-6057689790
Some findings
- MERGE inserts are no longer counted. Every sink of a copy-on-write MERGE
uses the copy-on-write writer, so the count is matched rows only.
- An insert-only MERGE now reports 0 where it used to report the inserted
rows, and so does any MERGE that matches nothing but inserts.
- The new test pins 3 for 2 updates, 2 inserts and 1 delete. As far as I
can tell, merge-on-read reports 5 for the same statement.
- The 0 is consumed elsewhere. IcebergColStatsWritePolicy.wroteNoRows()
turns it into SKIP, and insert-only MERGE is the one shape where column-stats
autogather is on (MergeSemanticAnalyzer:545).
HiveIcebergStorageHandler.getTable() also uses > 0 to decide whether to bypass
the table cache.
A possible fix: insert rows arrive with a NULL file path from the outer
join, so the writer could count them.
- count(*) OVER (PARTITION BY file_path) disables streaming. row_number()
alone streams. VectorPTFEvaluatorCountStar.streamsResult() is false, and the
row-mode path also rejects unbounded-following frames. Every matched row, at
full width, is now buffered per data file (and can spill) to produce a counter.
This needs a benchmark on a large DELETE, or a cheaper way to get the count.
- RECORDS_OUT_* changes meaning too. row_count backs both that counter and
TOTAL_TABLE_ROWS_WRITTEN, so the DAG summary will show matched rows instead of
records written. A separate LongWritable would avoid this.
- Tests cover only one data file. All three use a single INSERT into an
unpartitioned table, so summing across files and tasks is untested. Insert-only
MERGE, zero-match MERGE and a partitioned table are also missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
