[
https://issues.apache.org/jira/browse/IMPALA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18062282#comment-18062282
]
ASF subversion and git services commented on IMPALA-10540:
----------------------------------------------------------
Commit 6a70307092b51447ce13ae6546e6f2a978066997 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=6a7030709 ]
IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
JVM threads are being reused across JNI invocations. This means
if the interrupt flag is being set for a thread when the JNI call
returns, the next JNI call might get the interrupt (if it checks for
it).
It can be reproduced manually by adding
Thread.currentThread().interrupt() calls at the end of our methods
(e.g. JniFrontend.convertTable(), JniCatalog.updateCatalog()).
Doing so we can trigger errors we can see in IMPALA-12621, IMPALA-10633,
IMPALA-10924, IMPALA-10540, IMPALA-12261. All these issues have
a stack trace similar to the following:
W20260221 07:53:31.855443 1324125 DataStreamer.java:832] DataStreamer Exception
Java exception follows:
java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:477)
at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:781)
I20260221 07:53:31.863165 1312138 jni-util.cc:321]
fa4015e85de1494e:e1586d1500000000]
org.apache.iceberg.exceptions.RuntimeIOException: Failed to write json to file:
hdfs://localhost:20500/test-warehouse/lineitem_sixblocks_iceberg/metadata/00000-93a5f622-d41e-4ccb-87a4-eb26d9bd7a5c.metadata.json
at
org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:133)
at
org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:115)
at
org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:170)
at
org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadataIfRequired(BaseMetastoreTableOperations.java:160)
at
org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:173)
at
org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
at
org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:201)
at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:75)
at
org.apache.impala.catalog.iceberg.IcebergHiveCatalog.createTable(IcebergHiveCatalog.java:74)
at
org.apache.impala.util.MigrateTableUtil.migrateToIcebergTable(MigrateTableUtil.java:99)
at org.apache.impala.service.Frontend.convertTable(Frontend.java:1004)
at
org.apache.impala.service.JniFrontend.convertTable(JniFrontend.java:243)
I.e., something interrupted the thread when it tried to create a new
JSON file for an Iceberg table. Looking at the HDFS logs, the file
was created successfully:
org.apache.hadoop.hdfs.StateChange: DIR* completeFile: ...metadata.json
There is also no thread interruption logic in the code path of
JniFrontend.convertTable() that could explain the interruption.
So probably the best explanation to the above is:
* JNI thread gets interrupted in a way its interrupt flag remains set
* New JNI call on thread runs Iceberg code that checks interruption
* Above exception being thrown
Testing:
* tested manually by adding Thread.currentThread().interrupt() calls
at the end of JniCatalog/JniFronted methods.
Generated-by: Gemini Pro
Generated-by: Claude Sonnet 4.5
Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Reviewed-on: http://gerrit.cloudera.org:8080/24029
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> test_iceberg_inserts is flaky
> -----------------------------
>
> Key: IMPALA-10540
> URL: https://issues.apache.org/jira/browse/IMPALA-10540
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Reporter: Csaba Ringhofer
> Assignee: Zoltán Borók-Nagy
> Priority: Major
>
> Saw the following test failure while no related code was changed:
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/13270/testReport/junit/stress.test_insert_stress/TestInsertStress/test_iceberg_inserts_unique_database0_/
> {code}
> -- 2021-02-22 22:32:42,734 INFO Thread-88: Started query
> 4c4dddccf680c64f:de93794300000000
> -- executing against localhost:21000
> select * from test_iceberg_inserts_2d0d6e21.test_concurrent_inserts;
> -- executing against localhost:21002
> insert into table test_iceberg_inserts_2d0d6e21.test_concurrent_inserts
> values (2, 3);
> -- 2021-02-22 22:32:42,921 INFO Thread-89: Started query
> bc49486d3b332b46:09b479b000000000
> -- 2021-02-22 22:32:42,935 INFO Thread-92: Started query
> 2945b084593d6718:86af050700000000
> -- closing connection to: localhost:21000
> Traceback (most recent call last):
> File "/home/ubuntu/Impala/tests/stress/stress_util.py", line 34, in run
> return self.func(*self.args, **self.kwargs)
> File "/home/ubuntu/Impala/tests/stress/test_insert_stress.py", line 52, in
> _impala_role_concurrent_writer
> tbl_name, wid, insert_cnt))
> File "/home/ubuntu/Impala/tests/common/impala_connection.py", line 205, in
> execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 187, in
> execute
> handle = self.__execute_query(query_string.strip(), user=user)
> File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 365, in
> __execute_query
> self.wait_for_finished(handle)
> File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 386, in
> wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> ImpalaBeeswaxException: ImpalaBeeswaxException:
> Query aborted:RuntimeIOException: Failed to write json to file:
> hdfs://localhost:20500/test-warehouse/test_iceberg_inserts_2d0d6e21/test_iceberg_inserts_2d0d6e21/test_concurrent_inserts/metadata/fe9e49ab-7410-44f9-b84c-bfb897bc9c2f.metadata.json
> CAUSED BY: IOException: The stream is closed
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
