Hi
We have an application that writes parquet files. I am using the
AvroParquetWriter to write parquet files. While this piece of code works
fine in a Kerberos environment, it is failing when SSL is enabled in the
Hadoop cluster. So, I had modified the code to use the swebhdfs protocol
instead of the webhdfs and it is still failing with the following exception.
{
conf.set("hadoop.security.authentication", "kerberos");
UserGroupInformation.setConfiguration(conf);
ugi =
UserGroupInformation.loginUserFromKeytabAndReturnUGI(_user,_keytab) ;
try
{
ugi.doAs(new PrivilegedExceptionAction<Object>()
{
public Object run() throws IOException
{
fs = FileSystem.get(hdfsuri,conf) ;
if (_fileExistsAction == VAL_FILEEXISTS_OVERWRITE)
fs.delete(new Path(_fileName),false) ;
_writer = new AvroParquetWriter(new Path(hdfsuri),
_schema, _ParquetCompressionCodec, _ParquetBlockSize, _ParquetPageSize);
return null ;
}
}
);
hdfsuri in this case is of the format "swebhdfs://"+_host+":"+_port + "/" +
_fileName"
*The application is failing with the following exception :*
*===========================================*
Error :
Caused by
org.apache.hadoop.ipc.RemoteException(javax.ws.rs.WebApplicationException):
null
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:124)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:420)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:108)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.connect(WebHdfsFileSystem.java:596)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:674)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:524)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:554)
at
java.security.AccessController.doPrivileged(AccessController.java:488)
at javax.security.auth.Subject.doAs(Subject.java:572)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:550)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.create(WebHdfsFileSystem.java:1257)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:926)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:907)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
at parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:225)
at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:302)
at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:253)
at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:219)
at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:153)
at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:119)
at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:92)
at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:66)
at parquet.avro.AvroParquetWriter.<init>(AvroParquetWriter.java:54)
at
com.ibm.iis.jis.utilities.parquet.ParquetBuilder$1.run(ParquetBuilder.java:191)
at
java.security.AccessController.doPrivileged(AccessController.java:488)
at javax.security.auth.Subject.doAs(Subject.java:572)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at
com.ibm.iis.jis.utilities.parquet.ParquetBuilder.open(ParquetBuilder.java:182)
at
com.ibm.iis.jis.utilities.dochandler.impl.OutputBuilder.<init>(OutputBuilder.java:83)
at
com.ibm.iis.jis.utilities.dochandler.impl.Registrar.getBuilder(Registrar.java:331)
at
com.ibm.iis.jis.utilities.dochandler.impl.Registrar.getBuilder(Registrar.java:293)
at com.ibm.iis.cc.filesystem.FileSystem.getBuilder(FileSystem.java:2177)
at
com.ibm.iis.cc.filesystem.FileSystem.writeDelimitedFiles(FileSystem.java:1168)
at com.ibm.iis.cc.filesystem.FileSystem.writeFiles(FileSystem.java:922)
at com.ibm.iis.cc.filesystem.FileSystem.process(FileSystem.java:763)
at
com.ibm.is.cc.javastage.connector.CC_JavaAdapter.run(CC_JavaAdapter.java:443)
Any reason why the writing is failing with security exception ? I have
checked the Kerberos and SSL Debug logs, but there is no indication why it
is failing with security exception..
--
Srinivas
(*-*)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
You have to grow from the inside out. None can teach you, none can make you
spiritual.
-Narendra Nath Dutta(Swamy Vivekananda)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
