rdtr commented on code in PR #13560: URL: https://github.com/apache/gluten/pull/13560#discussion_r4543629484
########## cpp/core/jni/JavaException.cc: ########## @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "JavaException.h" + +#include <glog/logging.h> + +#include <stdexcept> +#include <string> + +namespace gluten { +namespace { + +// The describer class and result need two locals; reserve one spare slot. +constexpr jint kDescriptionLocalCapacity = 3; + +class LocalFrame { + public: + explicit LocalFrame(JNIEnv* env) : env_(env), pushed_(env->PushLocalFrame(kDescriptionLocalCapacity) == JNI_OK) {} + + ~LocalFrame() { + if (pushed_) { + env_->PopLocalFrame(nullptr); + } + } + + bool pushed() const { + return pushed_; + } + + private: + JNIEnv* env_; + bool pushed_; +}; + +std::string describe(JNIEnv* env, jthrowable throwable) { Review Comment: Could `describe` get a short comment on where its result ends up? Reading only this file, the message looks diagnostic-only, since `JNI_METHOD_END` rethrows the original and only falls back to `what()` when that fails. With the pinned Velox a throw inside an operator call is caught by `CALL_OPERATOR` and rethrown as `VELOX_FAIL("Operator::{} failed for [...]: {}", ..., e.what())`. https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/exec/Driver.cpp#L74-L84 https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/exec/Driver.cpp#L413-L414 #13561 and #13563 wrap the input-iterator and memory callbacks so the object survives that, but the `jol:`/`jni:` spill filesystem callbacks in JniFileSystem.cc, used when `spillFileSystem=heap-over-local` is set, aren't wrapped in any of the three PRs. On that path this string stays the only part of the Java exception that reaches Spark. The destructors there also log only `e.what()`. https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/velox/jni/JniFileSystem.cc#L83-L84 https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/velox/jni/JniFileSystem.cc#L164-L165 A line like "Used by what(). Paths that can't carry the JavaException itself, such as Velox operator wrapping and destructor logging, only see this text." would explain why it is built eagerly and why failures fall back to a placeholder. ########## cpp/core/jni/JavaException.cc: ########## @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "JavaException.h" + +#include <glog/logging.h> + +#include <stdexcept> +#include <string> + +namespace gluten { +namespace { + +// The describer class and result need two locals; reserve one spare slot. +constexpr jint kDescriptionLocalCapacity = 3; + +class LocalFrame { + public: + explicit LocalFrame(JNIEnv* env) : env_(env), pushed_(env->PushLocalFrame(kDescriptionLocalCapacity) == JNI_OK) {} + + ~LocalFrame() { + if (pushed_) { + env_->PopLocalFrame(nullptr); + } + } + + bool pushed() const { + return pushed_; + } + + private: + JNIEnv* env_; + bool pushed_; +}; + +std::string describe(JNIEnv* env, jthrowable throwable) { + const std::string prefix = "Error during calling Java code from native code: "; + const auto unavailable = [&](const char* reason) { + // Describing an exception must not replace the throwable we are transporting. + env->ExceptionClear(); + return prefix + "Java stack trace unavailable (" + reason + ")"; + }; + LocalFrame frame(env); + if (!frame.pushed()) { + return unavailable("PushLocalFrame failed"); + } + auto describer = env->FindClass("org/apache/gluten/exception/JniExceptionDescriber"); + if (describer == nullptr) { + return unavailable("JniExceptionDescriber lookup failed"); + } + auto method = env->GetStaticMethodID(describer, "describe", "(Ljava/lang/Throwable;)Ljava/lang/String;"); + if (method == nullptr) { + return unavailable("describe method lookup failed"); + } + auto description = static_cast<jstring>(env->CallStaticObjectMethod(describer, method, throwable)); + if (env->ExceptionCheck() || description == nullptr) { + return unavailable("describe method failed"); + } + const auto release = [&](const char* chars) { env->ReleaseStringUTFChars(description, chars); }; + std::unique_ptr<const char, decltype(release)> chars(env->GetStringUTFChars(description, nullptr), release); + if (!chars) { + return unavailable("GetStringUTFChars failed"); + } + return prefix + chars.get(); +} + +} // namespace + +struct JavaException::State { + State(JNIEnv* env, jthrowable throwable) { + if (throwable == nullptr) { + throw std::invalid_argument("JavaException requires a non-null throwable"); + } + if (env->GetJavaVM(&vm) != JNI_OK) { + throw std::runtime_error("Unable to get JavaVM for Java exception"); + } + ref = static_cast<jthrowable>(env->NewGlobalRef(throwable)); + if (ref == nullptr) { + throw std::runtime_error("Unable to create global reference for Java exception"); + } + } + + ~State() { + JNIEnv* env = nullptr; + auto status = vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION_1_8); + const bool detached = status == JNI_EDETACHED; + if (detached) { + status = vm->AttachCurrentThreadAsDaemon(reinterpret_cast<void**>(&env), nullptr); + } + if (status != JNI_OK) { + LOG(ERROR) << "Unable to attach thread to release Java exception global reference: " << status; + return; + } + env->DeleteGlobalRef(ref); + // Only detach if this destructor attached the thread; never detach a caller's JNI/HDFS thread. + if (detached && vm->DetachCurrentThread() != JNI_OK) { + LOG(ERROR) << "Unable to detach thread after releasing Java exception global reference"; + } + } + + JavaVM* vm{nullptr}; + jthrowable ref{nullptr}; + std::string message; +}; + +JavaException::JavaException(JNIEnv* env, jthrowable throwable) : state_(std::make_shared<State>(env, throwable)) { + try { + state_->message = describe(env, throwable); + } catch (const std::exception&) { + // Keep the captured throwable even if diagnostics fail. what() must not allocate a fallback. + env->ExceptionClear(); + } +} + +const char* JavaException::what() const noexcept { + return state_->message.empty() + ? "Error during calling Java code from native code: Java stack trace unavailable (native diagnostic failure)" + : state_->message.c_str(); +} + +void JavaException::throwToJava(JNIEnv* env) const noexcept { + if (env->Throw(state_->ref) != JNI_OK) { + LOG(ERROR) << "Unable to restore original Java exception"; + } +} + +bool JavaException::throwNestedToJava(JNIEnv* env, const std::exception& error) noexcept { + const auto* nested = dynamic_cast<const std::nested_exception*>(&error); + if (nested == nullptr || nested->nested_ptr() == nullptr) { + return false; + } + try { + std::rethrow_exception(nested->nested_ptr()); + } catch (const JavaException& original) { + original.throwToJava(env); + return true; + } catch (const std::exception& cause) { + return throwNestedToJava(env, cause); + } catch (...) { + // Not a Java exception. Leave the enclosing native error to the existing JNI fallback. + return false; + } +} + +void JavaException::check(JNIEnv* env) { + if (!env->ExceptionCheck()) { + return; + } + auto throwable = env->ExceptionOccurred(); + env->ExceptionClear(); + const auto release = [env](jthrowable ref) { env->DeleteLocalRef(ref); }; + std::unique_ptr<_jthrowable, decltype(release)> local(throwable, release); + // Guard construction only: successful carrier propagation must leave JNI clear. + auto error = [env, throwable]() { + try { + return JavaException(env, throwable); + } catch (...) { + if (!env->ExceptionCheck() && env->Throw(throwable) != JNI_OK) { Review Comment: Risk, only when memory is short. When the `JavaException` can't be constructed here (`make_shared<State>` throws or `NewGlobalRef` returns null), this line puts the original throwable back as a pending exception with `env->Throw(throwable)` and rethrows the C++ error. Normally the thread entered native code through a JNI method, so the C++ error unwinds to `JNI_METHOD_END`, which sees the pending exception, leaves it alone and returns to Java. For example on the input iterator path: 1. `ColumnarBatchOutIterator.nativeHasNext` enters native code under `JNI_METHOD_START`. https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/core/jni/JniWrapper.cc#L450-L462 2. Velox runs the task serially on that same thread, and the input operator calls `RowVectorStream::hasNext`. https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/velox/compute/WholeStageResultIterator.cc#L124-L138 https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/velox/operators/plannodes/RowVectorStream.cc#L51-L78 3. `JniColumnarBatchIterator::nextInternal` calls the Java iterator and then `checkException`, which is `JavaException::check`. https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/core/jni/JniCommon.cc#L300-L311 https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/core/jni/JniCommon.h#L56-L57 4. Construction fails and this line restores the throwable. https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/core/jni/JavaException.cc#L168-L176 5. The C++ error unwinds to the `std::exception` branch of `JNI_METHOD_END`. `env->ExceptionCheck()` is true there, so it neither restores a carrier nor throws `GlutenException`, and `nativeHasNext` returns `false`. https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/core/jni/JniError.h#L43-L51 The JVM then throws the original in Java when the native method returns, which also clears it from the thread. However, when the thread was started by Gluten's C++ code or by Velox, there is no Java frame below it to return to. Examples are the folly pools in `VeloxBackend` (spill, IO, broadcast build). https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/velox/compute/VeloxBackend.cc#L250-L274 Jobs submitted through `HookedExecutor` attach the thread before they run, and `SparkThreadInitializer::destroy` deliberately doesn't detach afterwards, so the thread stays attached for later jobs. https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/velox/compute/VeloxRuntime.cc#L167-L205 https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/core/jni/JniCommon.h#L611-L631 The `SparkAllocationListener` constructor comment already expects calls from IO threads. https://github.com/apache/gluten/blob/bb6b836892083724ed53421c06f8813ef3c3978e/cpp/core/jni/JniCommon.h#L403 On such a thread the C++ error is caught somewhere in C++, nothing clears the pending exception, and the thread goes back to its pool with it. The next job on that thread, possibly for a different task, makes a JNI call with an exception pending, and its `checkException` reports the old throwable as its own error. If `NewGlobalRef` fails with a pending `OutOfMemoryError`, line 172 keeps that one pending instead, with the same effect. Before this PR `checkException` always cleared the exception, so this is new. Could the restore happen only when a JNI entry is on the stack? For example `JNI_METHOD_START` could hold a small RAII guard that bumps a `thread_local` depth counter, and `check()` would restore only when the depth is above zero and clear otherwise. Pool threads that attached themselves have depth zero, so they keep the old clearing behavior, and a JNI method that doesn't use the macro would only lose the original in this rare case. If that's too much for this PR, always clearing here as `checkException` did before would also avoid the carry-over. `JNI_METHOD_END` would then report the C++ error as a `GlutenException`, so only the original throwable is lost, and only when memory is short. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
