A-little-bit-of-data opened a new issue, #6887: URL: https://github.com/apache/kyuubi/issues/6887
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug I have realized the relationship between users and groups through ldap, and I can see the corresponding information through the log, but I still can't get the corresponding spark sql task. ``` 2025-01-10 15:30:15 DEBUG LDAPGroupProvider:52 - mGroupQuery: (&(objectClass=posixGroup)(memberUid=evmli)) 2025-01-10 15:30:15 INFO LDAPGroupProvider:64 - User [evmli] belongs to groups: evm, predictive, evm_1, evm_1_1 2025-01-10 15:30:15 INFO ProcBuilder:64 - Creating evm's working directory at /opt/kyuubi/work/evm 2025-01-10 15:30:15 INFO Utils:64 - Loading Kyuubi properties from /opt/spark/conf/spark-defaults.conf 2025-01-10 15:30:16 INFO ProcBuilder:64 - Logging to /opt/kyuubi/work/evm/kyuubi-spark-sql-engine.log.0 2025-01-10 15:30:16 INFO EngineRef:64 - Launching engine: /opt/kyuubi/externals/spark-3.5.2-bin-hadoop3/bin/spark-submit \ --class org.apache.kyuubi.engine.spark.SparkSQLEngine \ --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \ --conf spark.kyuubi.client.ipAddress=xxxx \ --conf spark.kyuubi.client.version=1.10.1 \ --conf spark.kyuubi.engine.appMgrInfo=xxxx \ --conf spark.kyuubi.engine.engineLog.path=/opt/kyuubi/work/evm/kyuubi-spark-sql-engine.log.0 \ --conf spark.kyuubi.engine.share.level=GROUP \ --conf spark.kyuubi.engine.submit.time=1736494215818 \ --conf spark.kyuubi.engine.type=SPARK_SQL \ --conf spark.kyuubi.frontend.connection.url.use.hostname=false \ --conf spark.kyuubi.frontend.protocols=REST,THRIFT_BINARY \ --conf spark.kyuubi.ha.addresses=xxxx:2181 \ --conf spark.kyuubi.ha.engine.ref.id=56107026-fb7a-4ebd-8d6e-53e3af91ebbc \ --conf spark.kyuubi.ha.namespace=/kyuubi_1.10.1_GROUP_SPARK_SQL/evm/default \ --conf spark.kyuubi.kubernetes.namespace=dev \ --conf spark.kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=ALL \ --conf spark.kyuubi.kubernetes.terminatedApplicationRetainPeriod=PT1M \ --conf spark.kyuubi.metrics.enabled=true \ --conf spark.kyuubi.metrics.reporters= \ --conf spark.kyuubi.server.ipAddress=172.20.228.58 \ --conf spark.kyuubi.session.connection.url=172.20.228.58:10009 \ --conf spark.kyuubi.session.engine.check.interval=PT1M \ --conf spark.kyuubi.session.engine.idle.timeout=PT2M \ --conf spark.kyuubi.session.engine.trino.connection.catalog=s3dev \ --conf spark.kyuubi.session.engine.trino.connection.url=https://trino-ranger-dev \ --conf spark.kyuubi.session.group.ldap.based.dn=dc=dfg,dc=com \ --conf spark.kyuubi.session.group.ldap.bind.dn=cn=admin,dc=dfg,dc=com \ --conf spark.kyuubi.session.group.ldap.bind.password=ol_D5-mO7KWz.yg \ --conf spark.kyuubi.session.group.ldap.group.member.attr=memberUid \ --conf spark.kyuubi.session.group.ldap.group.name.attr=cn \ --conf spark.kyuubi.session.group.ldap.group.search.filter=(objectClass=posixGroup) \ --conf spark.kyuubi.session.group.ldap.url=ldap://openldap.dev:389 \ --conf spark.kyuubi.session.group.ldap.user.search.filter=(&(objectClass=person)(cn={0})) \ --conf spark.kyuubi.session.group.provider=ldap \ --conf spark.kyuubi.session.real.user=evmli \ --conf spark.app.name=kyuubi_GROUP_SPARK_SQL_evm_default_56107026-fb7a-4ebd-8d6e-53e3af91ebbc \ --conf spark.kubernetes.driver.label.kyuubi-unique-tag=56107026-fb7a-4ebd-8d6e-53e3af91ebbc \ --conf spark.kubernetes.driver.pod.name=kyuubi-group-spark-sql-evm-default-56107026-fb7a-4ebd-8d6e-53e3af91ebbc-driver \ --conf spark.kubernetes.executor.podNamePrefix=kyuubi-group-spark-sql-evm-default-56107026-fb7a-4ebd-8d6e-53e3af91ebbc \ --conf spark.user=evmli \ --conf spark.kubernetes.driverEnv.SPARK_USER_NAME=evm \ --conf spark.executorEnv.SPARK_USER_NAME=evm \ --proxy-user evm /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.10.1.jar 2025-01-10 15:30:17 DEBUG ClientCnxn:846 - Reading reply sessionid:0x100046a74831d25, packet:: clientPath:null serverPath:null finished:false header:: 8,12 replyHeader:: 8,118287032,-101 request:: '/kyuubi_1.10.1_GROUP_SPARK_SQL/evm/default,F response:: v{} 2025-01-10 15:30:17 DEBUG JDBCMetadataStore:52 - executeQuery sql: SELECT identifier,session_type,real_user,user_name,ip_address,kyuubi_instance,state,resource,class_name,request_name,request_conf,request_args,create_time,engine_type,cluster_manager,engine_open_time,engine_id,engine_name,engine_url,engine_state,engine_error,end_time,peer_instance_closed FROM metadata WHERE 1 = 1 AND session_type = ? AND state = ? AND kyuubi_instance = ? AND peer_instance_closed = ? ORDER BY key_id ASC LIMIT 2147483647 OFFSET 0, with params: BATCH, PENDING, 172.20.228.58:10099, true 2025-01-10 15:30:17 DEBUG JDBCMetadataStore:52 - executeQuery sql: SELECT identifier,session_type,real_user,user_name,ip_address,kyuubi_instance,state,resource,class_name,request_name,request_conf,request_args,create_time,engine_type,cluster_manager,engine_open_time,engine_id,engine_name,engine_url,engine_state,engine_error,end_time,peer_instance_closed FROM metadata WHERE 1 = 1 AND session_type = ? AND state = ? AND kyuubi_instance = ? AND peer_instance_closed = ? ORDER BY key_id ASC LIMIT 2147483647 OFFSET 0, with params: BATCH, RUNNING, 172.20.228.58:10099, true 2025-01-10 15:30:18 DEBUG ClientCnxn:846 - Reading reply sessionid:0x100046a74831d25, packet:: clientPath:null serverPath:null finished:false header:: 9,12 replyHeader:: 9,118287034,-101 request:: '/kyuubi_1.10.1_GROUP_SPARK_SQL/evm/default,F response:: v{} 2025-01-10 15:30:19 DEBUG ClientCnxn:846 - Reading reply sessionid:0x100046a74831d25, packet:: clientPath:null serverPath:null finished:false header:: 10,12 replyHeader:: 10,118287034,-101 request:: '/kyuubi_1.10.1_GROUP_SPARK_SQL/evm/default,F response:: v{} 2025-01-10 15:30:20 DEBUG ClientCnxn:846 - Reading reply sessionid:0x100046a74831d25, packet:: clientPath:null serverPath:null finished:false header:: 11,12 replyHeader:: 11,118287034,-101 request:: '/kyuubi_1.10.1_GROUP_SPARK_SQL/evm/default,F response:: v{} 2025-01-10 15:30:20 DEBUG KyuubiServer:52 - OperationHandle [785a9f40-1ead-4b63-af37-7bf18768b6c7]: Long polling timed out, null 2025-01-10 15:30:20 DEBUG TSaslTransport:466 - writing data length: 79 2025-01-10 15:30:20 DEBUG TSaslTransport:427 - SERVER: reading data length: 117 2025-01-10 15:30:20 DEBUG KyuubiTBinaryFrontendService:52 - TFetchResultsReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:78 5A 9F 40 1E AD 4B 63 AF 37 7B F1 87 68 B6 C7, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:UNKNOWN, hasResultSet:false), orientation:FETCH_NEXT, maxRows:1000, fetchType:1) 2025-01-10 15:30:20 DEBUG TSaslTransport:466 - writing data length: 67 2025-01-10 15:30:20 DEBUG TSaslTransport:427 - SERVER: reading data length: 100 2025-01-10 15:30:20 DEBUG KyuubiTBinaryFrontendService:52 - TGetOperationStatusReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:78 5A 9F 40 1E AD 4B 63 AF 37 7B F1 87 68 B6 C7, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:UNKNOWN, hasResultSet:false)) 2025-01-10 15:30:21 DEBUG ClientCnxn:769 - Got notification sessionid:0x100046a74831d25 2025-01-10 15:30:21 DEBUG ClientCnxn:791 - Got WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/kyuubi_1.10.1_GROUP_SPARK_SQL_lock/evm/default/leases for sessionid 0x100046a74831d25 2025-01-10 15:30:21 DEBUG ClientCnxn:846 - Reading reply sessionid:0x100046a74831d25, packet:: clientPath:null serverPath:null finished:false header:: 12,2 replyHeader:: 12,118287037,0 request:: '/kyuubi_1.10.1_GROUP_SPARK_SQL_lock/evm/default/leases/_c_3fa241a2-039c-41b2-a6d3-0edd47190f83-lease-0000000009,-1 response:: null 2025-01-10 15:30:21 DEBUG CuratorFrameworkImpl:282 - Closing 2025-01-10 15:30:21 INFO CuratorFrameworkImpl:821 - backgroundOperationsLoop exiting 2025-01-10 15:30:21 DEBUG CuratorZookeeperClient:199 - Closing 2025-01-10 15:30:21 DEBUG ConnectionState:115 - Closing 2025-01-10 15:30:21 DEBUG ZooKeeper:682 - Closing session: 0x100046a74831d25 2025-01-10 15:30:21 DEBUG ClientCnxn:1370 - Closing client for session: 0x100046a74831d25 2025-01-10 15:30:21 DEBUG ClientCnxn:846 - Reading reply sessionid:0x100046a74831d25, packet:: clientPath:null serverPath:null finished:false header:: 13,-11 replyHeader:: 13,118287038,0 request:: null response:: null 2025-01-10 15:30:21 DEBUG ClientCnxn:1354 - Disconnecting client for session: 0x100046a74831d25 2025-01-10 15:30:22 DEBUG ClientCnxn:1146 - An exception was thrown while closing send thread for session 0x100046a74831d25 : Unable to read additional data from server sessionid 0x100046a74831d25, likely server has closed socket 2025-01-10 15:30:22 INFO ZooKeeper:693 - Session: 0x100046a74831d25 closed 2025-01-10 15:30:22 INFO ClientCnxn:522 - EventThread shut down for session: 0x100046a74831d25 2025-01-10 15:30:22 INFO LaunchEngine:64 - Processing evmli's query[785a9f40-1ead-4b63-af37-7bf18768b6c7]: RUNNING_STATE -> ERROR_STATE, time taken: 6.254 seconds 2025-01-10 15:30:22 INFO OperationAuditLogger:64 - operation=785a9f40-1ead-4b63-af37-7bf18768b6c7 opType=LaunchEngine state=ERROR user=evmli session=56107026-fb7a-4ebd-8d6e-53e3af91ebbc 2025-01-10 15:30:22 DEBUG KyuubiServer:52 - OperationHandle [785a9f40-1ead-4b63-af37-7bf18768b6c7]: The background operation was aborted, org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default.svc:443/api/v1/namespaces/dev/pods. Message: Pod "kyuubi-group-spark-sql-evm-default-56107026-fb7a-4ebd-8d6e-53e3af91ebbc-driver" is invalid: spec.containers[0].resources.requests: Invalid value: "1": must be less than or equal to cpu limit. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.containers[0].resources.requests, message=Invalid value: "1": must be less than or equal to cpu limit, reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Pod, name=kyuubi-group-spark-sql-evm-default-56107026-fb7a-4ebd-8d6e-53e3af91ebbc-driver, retryAfterSeconds=null, uid=null, ad ditionalProperties={}), kind=Status, message=Pod "kyuubi-group-spark-sql-evm-default-56107026-fb7a-4ebd-8d6e-53e3af91ebbc-driver" is invalid: spec.containers[0].resources.requests: Invalid value: "1": must be less than or equal to cpu limit, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}). at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:518 ``` I also looked at the source code related to the engine startup, which should be like this` case GROUP => groupProvider.primaryGroup(sessionUser, conf.getAll.asJava) `Press the first one of the returned group, then I should submit the task according to evm here and configure the relevant information，I also tried to submit using admin, and it worked because the group name and user name of admin are the same. I want to know why the evmli user cannot use the group name evm to start the spark SQL task. ``` ___admin___.spark.app.name=sparksql-admin ___admin___.spark.executor.instances=1 ___admin___.spark.driver.cores=1 ___admin___.spark.executor.cores=1 ___admin___.spark.kubernetes.driver.limit.cores=1 ___admin___.spark.kubernetes.executor.limit.cores=1 ___admin___.spark.driver.memory=1g ___admin___.spark.executor.memory=1g ___evm___.spark.app.name=sparksql-admin ___evm___.spark.executor.instances=1 ___evm___.spark.driver.cores=1 ___evm___.spark.executor.cores=1 ___evm___.spark.kubernetes.driver.limit.cores=1 ___evm___.spark.kubernetes.executor.limit.cores=1 ___evm___.spark.driver.memory=1g ___evm___.spark.executor.memory=1g ``` ### Affects Version(s) 1.10.1 ### Kyuubi Server Log Output _No response_ ### Kyuubi Engine Log Output _No response_ ### Kyuubi Server Configurations _No response_ ### Kyuubi Engine Configurations _No response_ ### Additional context _No response_ ### Are you willing to submit PR? - [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. - [ ] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@kyuubi.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: notifications-unsubscr...@kyuubi.apache.org For additional commands, e-mail: notifications-h...@kyuubi.apache.org
