manoj-kumar-tg opened a new issue, #5831: URL: https://github.com/apache/incubator-gluten/issues/5831
### Backend VL (Velox) ### Bug description ``` zetta_spark3@dc78cde26b0c:~$ ./spark/bin/spark-sql --master local --conf spark.executor.instances=3 --conf spark.executor.cores=5 --conf spark.memory.offHeap.enabled=true --conf spark.executor.memory=40g --conf spark.driver.cores=5 --driver-memory 50g --conf spark.plugins=io.glutenproject.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=20g --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.backend.velox.SplitPreloadPerDriver=0 --database tpcds_100g -f tpcds-data-gen/spark-sql-perf/src/main/resources/tpcds_2_4/q3.sql Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 24/05/17 15:02:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 24/05/17 15:02:17 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist 24/05/17 15:02:17 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist 24/05/17 15:02:19 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0 24/05/17 15:02:19 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore [email protected] 24/05/17 15:02:19 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException Spark master: local, Application Id: local-1715938336665 24/05/17 15:02:22 WARN SparkShimProvider: Spark runtime version 3.3.2 is not matched with Gluten's fully tested version 3.3.1 E0517 15:02:23.888530 1561776 Exceptions.h:69] Line: /home/zetta/incubator-gluten-1.1.1/ep/build-velox/build/velox_ep/./velox/common/memory/MemoryPool.h:881, Function:sanityCheckLocked, Expression: Bad memory usage track state: Memory Pool[op.0.0.0.TableScan LEAF root[WholeStageIterator_root] parent[node.0] MALLOC track-usage thread-safe]<unlimited max capacity unlimited capacity used 16777216.00TB available 2.69KB reservation [used 16777216.00TB, reserved 0B, min 0B] counters [allocs 566, frees 565, reserves 0, releases 1, collisions 0])>, Source: RUNTIME, ErrorCode: INVALID_STATE terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError' what(): Exception: VeloxRuntimeError Error Source: RUNTIME Error Code: INVALID_STATE Reason: Bad memory usage track state: Memory Pool[op.0.0.0.TableScan LEAF root[WholeStageIterator_root] parent[node.0] MALLOC track-usage thread-safe]<unlimited max capacity unlimited capacity used 16777216.00TB available 2.69KB reservation [used 16777216.00TB, reserved 0B, min 0B] counters [allocs 566, frees 565, reserves 0, releases 1, collisions 0])> Retriable: False Function: sanityCheckLocked File: /home/zetta/incubator-gluten-1.1.1/ep/build-velox/build/velox_ep/./velox/common/memory/MemoryPool.h Line: 881 Stack trace: # 0 facebook::velox::VeloxException::VeloxException(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >) # 1 void facebook::velox::detail::veloxCheckFail<facebook::velox::VeloxRuntimeError, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(facebook::velox::detail::VeloxCheckFailArgs const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) # 2 facebook::velox::memory::MemoryPoolImpl::releaseThreadSafe(unsigned long, bool) # 3 facebook::velox::FlatVector<facebook::velox::StringView>::~FlatVector() # 4 facebook::velox::DictionaryVector<facebook::velox::StringView>::~DictionaryVector() # 5 facebook::velox::RowVector::~RowVector() # 6 void __gnu_cxx::new_allocator<gluten::VeloxColumnarBatch>::destroy<gluten::VeloxColumnarBatch>(gluten::VeloxColumnarBatch*) # 7 std::_Hashtable<long, std::pair<long const, std::shared_ptr<void> >, std::allocator<std::pair<long const, std::shared_ptr<void> > >, std::__detail::_Select1st, std::equal_to<long>, std::hash<long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_erase(unsigned long, std::__detail::_Hash_node_base*, std::__detail::_Hash_node<std::pair<long const, std::shared_ptr<void> >, false>*) # 8 gluten::ObjectStore::release(long) # 9 Java_io_glutenproject_columnarbatch_ColumnarBatchJniWrapper_close # 10 0x00007ff1ad017da6 # 11 0x00007ff1ad007fd3 # 12 0x00007ff1ad007fd3 # 13 0x00007ff1ad007fd3 # 14 0x00007ff1ad007fd3 # 15 0x00007ff1ad007fd3 # 16 0x00007ff1ad007d5f # 17 0x00007ff1aeff030f ``` ### Spark version Spark-3.3.x ### Spark configurations ./spark/bin/spark-sql --verbose --master local --conf spark.executor.instances=3 --conf spark.executor.cores=5 --conf spark.executor.memory=40g --conf spark.driver.cores=5 --driver-memory 50g --conf spark.plugins=io.glutenproject.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=20g --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager - ### System information System information zetta@dc78cde26b0c:~$ bash incubator-gluten-1.1.1/dev/info.sh Velox System Info v0.0.2 Commit: Not in a git repo. CMake Version: 3.22.1 System: Linux-5.15.0-107-generic Arch: x86_64 C++ Compiler: /opt/AMD/aocc-compiler-4.2.0/bin/clang++ C++ Compiler Version: 16.0.3 C Compiler: /opt/AMD/aocc-compiler-4.2.0/bin/clang C Compiler Version: 16.0.3 CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt ### Relevant logs _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
