chrajeshbabu commented on issue #14552:
URL: https://github.com/apache/pinot/issues/14552#issuecomment-2533385724
Able to make the job gets started by excluding the dependency causing
org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:472) loading
from pinot jar which leads to this issue. Found other problems the way the
dependencies get's pulled before starting the mappers.
During the job initialisation pushing the dependencies tar ball to HDFS path.
```
File pluginsTarGzFile = new File(PINOT_PLUGINS_TAR_GZ);
try {
File[] files = validPluginDirectories.toArray(new File[0]);
TarCompressionUtils.createCompressedTarFile(files, pluginsTarGzFile);
// Copy to staging directory
Path cachedPluginsTarball = new Path(stagingDirURI.toString(),
SegmentGenerationUtils.PINOT_PLUGINS_TAR_GZ);
outputDirFS.copyFromLocalFile(pluginsTarGzFile,
cachedPluginsTarball.toUri());
job.addCacheFile(cachedPluginsTarball.toUri());
} catch (Exception e) {
LOGGER.error("Failed to tar plugins directories and upload to staging
dir", e);
throw new RuntimeException(e);
}
```
But in the mappers expecting the tar ball to be available at the local file
system which are not downloaded from the HDFS before starting the mappers so
the job fails with CNFE.
```
File localPluginsTarFile = new File(PINOT_PLUGINS_TAR_GZ);
if (localPluginsTarFile.exists()) {
File pluginsDirFile =
Files.createTempDirectory(PINOT_PLUGINS_DIR).toFile();
try {
TarCompressionUtils.untar(localPluginsTarFile, pluginsDirFile);
} catch (Exception e) {
LOGGER.error("Failed to untar local Pinot plugins tarball file
[{}]", localPluginsTarFile, e);
throw new RuntimeException(e);
}
LOGGER.info("Trying to set System Property: {}={}",
PLUGINS_DIR_PROPERTY_NAME, pluginsDirFile.getAbsolutePath());
System.setProperty(PLUGINS_DIR_PROPERTY_NAME,
pluginsDirFile.getAbsolutePath());
String pluginsIncludes = _jobConf.get(PLUGINS_INCLUDE_PROPERTY_NAME);
if (pluginsIncludes != null) {
LOGGER.info("Trying to set System Property: {}={}",
PLUGINS_INCLUDE_PROPERTY_NAME, pluginsIncludes);
System.setProperty(PLUGINS_INCLUDE_PROPERTY_NAME, pluginsIncludes);
}
LOGGER.info("Pinot plugins System Properties are set at [{}], plugins
includes [{}]",
System.getProperty(PLUGINS_DIR_PROPERTY_NAME),
System.getProperty(PLUGINS_INCLUDE_PROPERTY_NAME));
} else {
LOGGER.warn("Cannot find local Pinot plugins directory at [{}]",
localPluginsTarFile.getAbsolutePath());
}
```
Fixing this to make it work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
