sadikovi commented on PR #36366:
URL: https://github.com/apache/spark/pull/36366#issuecomment-1110364545
The implications of the bug are the following:
- All missing columns are treated as null and either report wrong results or
crash JVM.
- All fully null columns have a similar behaviour.
Examples of errors:
```
[info]
[info] == Results ==
[info] !== Correct Answer - 3 == == Spark Answer - 3 ==
[info] !struct<> struct<_1:struct<_3:int,_4:bigint>>
[info] ![null] [[336920597,0]]
[info] ![null] [[827474256,0]]
[info] ![null] [[872417200,1447062946291269968]]
(QueryTest.scala:243)
```
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007ff3dd03ec6e, pid=19511, tid=0x00007ff2b39db700
#
# JRE version: OpenJDK Runtime Environment (8.0_292-b10) (build
1.8.0_292-8u292-b10-0ubuntu1~18.04-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.292-b10 mixed mode linux-amd64
compressed oops)
# Problematic frame:
# J 7824 C1
org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.getLong(I)J (41
bytes) @ 0x00007ff3dd03ec6e [0x00007ff3dd03eb60+0x10e]
#
# Failed to write core dump. Core dumps have been disabled. To enable core
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/ivan.sadikov/spark-oss/sql/core/hs_err_pid19511.log
Compiled method (c1) 9692 7823 3
org.apache.spark.sql.vectorized.ColumnarRow::getLong (16 bytes)
total in heap [0x00007ff3dd03cd90,0x00007ff3dd03d2f8] = 1384
relocation [0x00007ff3dd03ceb8,0x00007ff3dd03cf08] = 80
main code [0x00007ff3dd03cf20,0x00007ff3dd03d160] = 576
stub code [0x00007ff3dd03d160,0x00007ff3dd03d218] = 184
oops [0x00007ff3dd03d218,0x00007ff3dd03d220] = 8
metadata [0x00007ff3dd03d220,0x00007ff3dd03d228] = 8
scopes data [0x00007ff3dd03d228,0x00007ff3dd03d268] = 64
scopes pcs [0x00007ff3dd03d268,0x00007ff3dd03d2d8] = 112
dependencies [0x00007ff3dd03d2d8,0x00007ff3dd03d2e0] = 8
nul chk table [0x00007ff3dd03d2e0,0x00007ff3dd03d2f8] = 24
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
