[
https://issues.apache.org/jira/browse/SPARK-35747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod KC updated SPARK-35747:
-----------------------------
Description:
In a secure Yarn cluster where HBase service is down, even if the spark
application is not using HBase, during the application submit stage,
HBaseDelegationTokenProvider prints full Exception Stack trace and it causes a
noisy warning.
{code:java}
WARN security.HBaseDelegationTokenProvider: Failed to get token from service
hbase
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokensWithHBaseConn(HBaseDelegationT
okenProvider.scala:93)
more than 100 line exception stack trace{code}
Also, Application submission taking more time as
`HBaseDelegationTokenProvider.obtainDelegationTokensWithHBaseConn` retries the
connection to HBase master multiple times before it gives up. This slows down
the application submission steps. Hence, if HBase is not used in the user
Application, it is better to suggest user to disable HBase Delegation Token
generation.
ie, spark.security.credentials.hbase.enabled=false
was:
In a secure Yarn cluster where HBase service is down, even if the spark
application is not using HBase, during the application submit stage,
HBaseDelegationTokenProvider prints full Exception Stack trace and it causes a
noisy warning.
{code:java}
WARN security.HBaseDelegationTokenProvider: Failed to get token from service
hbase
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokensWithHBaseConn(HBaseDelegationT
okenProvider.scala:93)
{code}
Also, Application submission taking more time as
`HBaseDelegationTokenProvider.obtainDelegationTokensWithHBaseConn` retries the
connection to HBase master multiple times before it gives up. This slows down
the application submission steps. Hence, if HBase is not used in the user
Application, it is better to suggest user to disable HBase Delegation Token
generation.
ie, spark.security.credentials.hbase.enabled=false
> Avoid printing full Exception stack trace, if HBase service is not running in
> a secure cluster
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-35747
> URL: https://issues.apache.org/jira/browse/SPARK-35747
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0, 3.1.0, 3.1.2
> Reporter: Vinod KC
> Priority: Minor
>
> In a secure Yarn cluster where HBase service is down, even if the spark
> application is not using HBase, during the application submit stage,
> HBaseDelegationTokenProvider prints full Exception Stack trace and it causes
> a noisy warning.
>
> {code:java}
> WARN security.HBaseDelegationTokenProvider: Failed to get token from service
> hbase
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokensWithHBaseConn(HBaseDelegationT
> okenProvider.scala:93)
> more than 100 line exception stack trace{code}
> Also, Application submission taking more time as
> `HBaseDelegationTokenProvider.obtainDelegationTokensWithHBaseConn` retries
> the connection to HBase master multiple times before it gives up. This slows
> down the application submission steps. Hence, if HBase is not used in the
> user Application, it is better to suggest user to disable HBase Delegation
> Token generation.
> ie, spark.security.credentials.hbase.enabled=false
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
