touchida opened a new pull request, #4964:
URL: https://github.com/apache/kyuubi/pull/4964
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in
https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title,
e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
It resolves https://github.com/apache/kyuubi/issues/4951.
It would be also possible to resolve it by adding log4j 1 slf4j binding
(e.g., adding `slf4j-api-1.7.30.jar` and `slf4j-log4j12-1.7.30`) to the Spark
jars directory, but it would be helpful if Kyuubi could support reload4j to
avoid log4j 1's security vulnerabilities.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Assume
- we use Spark SQL engine in YARN cluster deploy mode;
- we use Hadoop-free Spark 3.2.4 and provides the classpath of Hadoop in
YARN cluster, which uses reload4j, with `SPARK_DIST_CLASSPATH`.
Before applying the patch, Hive beeline shows operation logs only from
Kyuubi server, as follows.
```
0: jdbc:hive2://some_kyuubi_host> select * from some_database.some_table;
2023-06-14 22:55:35.584 INFO
org.apache.kyuubi.credentials.HadoopCredentialsManager: Send new credentials
with epoch 0 to SQL engine through session 5409e4cb-abe0-457f-b3c8-c3f1c5467c60
2023-06-14 22:55:35.604 INFO
org.apache.kyuubi.credentials.HadoopCredentialsManager: Update session
credentials epoch from -1 to 0
2023-06-14 22:55:35.662 INFO org.apache.kyuubi.operation.ExecuteStatement:
Processing some_user's query[013b8294-b395-4e3d-945e-3594b0603f83]:
PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
2023-06-14 22:55:38.520 INFO org.apache.kyuubi.operation.ExecuteStatement:
Query[013b8294-b395-4e3d-945e-3594b0603f83] in FINISHED_STATE
2023-06-14 22:55:38.520 INFO org.apache.kyuubi.operation.ExecuteStatement:
Processing some_user's query[013b8294-b395-4e3d-945e-3594b0603f83]:
RUNNING_STATE -> FINISHED_STATE, time taken: 2.857 seconds
+-----+
| id |
+-----+
| 1 |
+-----+
1 row selected (3.113 seconds)
```
After applying the patch, Hive beeline shows operation logs from both Kyuubi
server and Spark SQL engine, as follows.
```
0: jdbc:hive2://some_kyuubi_host> select * from some_database.some_table;
2023-06-14 23:05:57.424 INFO
org.apache.kyuubi.credentials.HadoopCredentialsManager: Send new credentials
with epoch 0 to SQL engine through session 146aa800-6047-471e-a480-5683197a84cd
2023-06-14 23:05:57.453 INFO
org.apache.kyuubi.credentials.HadoopCredentialsManager: Update session
credentials epoch from -1 to 0
2023-06-14 23:05:57.512 INFO org.apache.kyuubi.operation.ExecuteStatement:
Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]:
PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
23/06/14 23:05:57 INFO operation.ExecuteStatement: Processing some_user's
query[1c209bea-6f62-4854-acdc-a1945867a65d]: PENDING_STATE -> RUNNING_STATE,
statement:
select * from some_database.some_table
23/06/14 23:05:57 INFO operation.ExecuteStatement:
Spark application name:
kyuubi_CONNECTION_SPARK_SQL_some_user_146aa800-6047-471e-a480-5683197a84cd
application ID: application_1686295570868_0065
application web UI: some_yarn_application_url
master: yarn
deploy mode: cluster
version: 3.2.4
Start time: 2023-06-14T23:05:02.490
User: some_user
(omitted)
2023-06-14 23:06:00.285 INFO org.apache.kyuubi.operation.ExecuteStatement:
Query[1c209bea-6f62-4854-acdc-a1945867a65d] in FINISHED_STATE
2023-06-14 23:06:00.286 INFO org.apache.kyuubi.operation.ExecuteStatement:
Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]:
RUNNING_STATE -> FINISHED_STATE, time taken: 2.774 seconds
+-----+
| id |
+-----+
| 1 |
+-----+
1 row selected (3.156 seconds)
```
- [ ] [Run
test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests)
locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
