[
https://issues.apache.org/jira/browse/AMBARI-25987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robin Zuo updated AMBARI-25987:
-------------------------------
Description:
Currently, when run spark tasks, spark will report cannot load hadoop native
libraries.
Also, the hadoop short-circuit read feature cannot be applied. For better
performance, add hadoop native libraries support.
Some WARN log as following:
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
23/08/15 13:43:22 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Screen shot:
!image-2023-08-15-15-45-26-664.png|width=924,height=107!
...
23/08/15 13:43:25 INFO Client: Requesting a new application from cluster with 3
NodeManagers
23/08/15 13:43:26 WARN DomainSocketFactory: The short-circuit local reads
feature cannot be used because libhadoop cannot be loaded.
23/08/15 13:43:26 INFO Configuration: resource-types.xml not found
Screen shot:
!image-2023-08-15-15-46-43-947.png|width=780,height=102!
was:
Currently, when run spark tasks, spark will report cannot load hadoop native
libraries.
Also, the hadoop short-circuit read feature cannot be applied. For better
performance, add hadoop native libraries support.
Some WARN log as following:
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
23/08/15 13:43:22 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
...
23/08/15 13:43:25 INFO Client: Requesting a new application from cluster with 3
NodeManagers
23/08/15 13:43:26 WARN DomainSocketFactory: The short-circuit local reads
feature cannot be used because libhadoop cannot be loaded.
23/08/15 13:43:26 INFO Configuration: resource-types.xml not found
> Enable Spark to use hadoop native libraries for better performance
> ------------------------------------------------------------------
>
> Key: AMBARI-25987
> URL: https://issues.apache.org/jira/browse/AMBARI-25987
> Project: Ambari
> Issue Type: Improvement
> Components: ambari-server
> Reporter: Robin Zuo
> Assignee: Robin Zuo
> Priority: Major
> Attachments: image-2023-08-15-15-45-26-664.png,
> image-2023-08-15-15-46-43-947.png
>
>
> Currently, when run spark tasks, spark will report cannot load hadoop native
> libraries.
> Also, the hadoop short-circuit read feature cannot be applied. For better
> performance, add hadoop native libraries support.
> Some WARN log as following:
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 23/08/15 13:43:22 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
>
> Screen shot:
> !image-2023-08-15-15-45-26-664.png|width=924,height=107!
> ...
> 23/08/15 13:43:25 INFO Client: Requesting a new application from cluster with
> 3 NodeManagers
> 23/08/15 13:43:26 WARN DomainSocketFactory: The short-circuit local reads
> feature cannot be used because libhadoop cannot be loaded.
> 23/08/15 13:43:26 INFO Configuration: resource-types.xml not found
> Screen shot:
> !image-2023-08-15-15-46-43-947.png|width=780,height=102!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
