Re: datatorrent gateway application package problem

Wed, 28 Sep 2016 14:57:06 -0700

Check namenode logs. Are there any exceptions/errors raised at the same time as the gateway exception? 

On 9/28/16 14:43, d...@softiron.co.uk wrote:

On 2016-09-28 14:09, David Yan wrote:

dtgateway does not use the native libraries.

I just tried a clean install of RTS 3.5.0 community edition on plain
vanilla hadoop 2.7.3 on ubuntu 14.04 and was able to launch the pi
demo from the web interface. I was able to do the same even with the
native libraries removed.


Thanks.  I thought it was a long shot.


What is your fs.defaultFS property in hadoop? Can you verify that it
points to your hdfs (hdfs://namenode:9000/) ?


In $HADOOP_CONF_DIR/hdfs-site.xml, fs.defaultFS is hdfs://namenode:9000
Note also that if I try to give a URL to the dtgateway Installation Wizard 
that isn't the namenode host/port, it complains immediately, instead of
proceeding to the next page.


Unfortunately I don't have access to an ARM64 so any finding you post
here would be helpful.
I'm not surprised. A classic chicken-and-egg problem. There aren't a lot of machines around, so the software doesn't get a lot of attention, so people don't buy machines. 




David

On Wed, Sep 28, 2016 at 11:04 AM, <d...@softiron.co.uk> wrote:


I had a thought of what might be the problem:
the hadoop native library doesn't build on SUSE/SLES ARM64,
so I'm running with the pure Java library.

Is it possible that the dtgateway is trying to use the native
library
directly?

Thanks,
-david

On 2016-09-26 14:45, David Yan wrote:

Do you see any exception stacktrace in the log when this error
occurred:

Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent, expected:
file:///

David

On Mon, Sep 26, 2016 at 2:39 PM, <d...@softiron.co.uk> wrote:

From    David Yan <da...@datatorrent.com>
Date    Sat 00:12

Can you please provide any exception stacktrace in the
dtgateway.log file when that happens?

I reran the dtgateway installation wizard, which failed (same as
before) with error msg:

| DFS directory cannot be written to with error message "Mkdirs
failed to create /user/dtadmin/datatorrent (exists=false,
cwd=file:/opt/datatorrent/releases/3.4.0)"

The corresponding exception in dtgateway.log is:

| 2016-09-26 13:43:54,767 ERROR com.datatorrent.gateway.I: DFS
Directory cannot be written to with exception:
| java.io.IOException: Mkdirs failed to create
/user/dtadmin/datatorrent (exists=false,
cwd=file:/opt/datatorrent/releases/3.4.0)
|     at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455) 


|     at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440) 


|     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
|     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
|     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
|     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
|     at
com.datatorrent.stram.client.FSAgent.createFile(FSAgent.java:77)
|     at com.datatorrent.gateway.I.h(gc:324)
|     at com.datatorrent.gateway.I.h(gc:284)
|     at
com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:136)
|     at
com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:171)
|     at
com.datatorrent.gateway.resources.ws.v2.ConfigResource.setConfigProperty(fc:34) 


|     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
|     at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 


|     at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 


|     at java.lang.reflect.Method.invoke(Method.java:498)
|     at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) 


|     at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) 


|     at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) 


|     at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) 


|     at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) 


|     at
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134) 


|     at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) 


|     at
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134) 


|     at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) 


|     at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) 


|     at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) 


|     at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) 


|     at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) 


|     at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) 


|     at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) 


|     at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) 


|     at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) 


|     at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) 


|     at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) 


|     at
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
|     at

 org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)


|     at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:457) 


|     at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) 


|     at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) 


|     at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) 


|     at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) 


|     at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) 


|     at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) 


|     at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) 


|     at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) 


|     at org.eclipse.jetty.server.Server.handle(Server.java:368)
|     at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) 


|     at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953) 


|     at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014) 


|     at
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
|     at

 org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)


|     at
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) 


|     at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628) 


|     at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) 


|     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) 


|     at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) 


|     at java.lang.Thread.run(Thread.java:745)
| 2016-09-26 13:43:54,772 INFO
com.datatorrent.gateway.resources.ws.v2.WSResource: Caught exception
in processing web service:
com.datatorrent.stram.client.DTConfiguration$ConfigException: DFS
directory cannot be written to with error message "Mkdirs failed to
create /user/dtadmin/datatorrent (exists=false,
cwd=file:/opt/datatorrent/releases/3.4.0)"

Also, when I shutdown the gateway, I get a different exception in
dtgateway.log:

| 2016-09-26 13:40:44,366 INFO com.datatorrent.gateway.DTGateway:
Shutting down
| 2016-09-26 13:40:44,427 ERROR com.datatorrent.gateway.I: DFS
Directory cannot be written to with exception:
| java.io.IOException: Filesystem closed
|     at
org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808)
|     at
org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2041)
|     at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:707) 


|     at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:703) 


|     at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) 


|     at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:714) 


|     at
com.datatorrent.stram.client.FSAgent.deleteFile(FSAgent.java:94)
|     at com.datatorrent.gateway.I.h(gc:30)
|     at com.datatorrent.gateway.l.K(pc:158)
|     at com.datatorrent.gateway.l.h(pc:65)
|     at com.datatorrent.gateway.K.run(yc:156)
|     at java.lang.Thread.run(Thread.java:745)
| 2016-09-26 13:40:44,444 INFO com.datatorrent.gateway.DTGateway:
Shutdown complete

Thanks much.
-david

On Tue, Sep 13, 2016 at 4:06 PM, <d...@softiron.co.uk> wrote:

I have the dtgateway running (community edition) with
apex/malhar 3.4 and hadoop 2.7.4 on SuSE SLES 12.1 on an ARM64
server (aarch64).

In the Installation Wizard, I set the DFS location to:

hdfs://namenode:9000/user/dtadmin/datatorrent

and the gateway saves the hadoop configuration and restarts
successfully, but  I get an error:

Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent,
expected: file:///

the 'service dtgateway status' command says it's running.
The log file has the same error as above, but nothing else
useful.

Is there a manual way to move the demo apps into the
gateway?

If I ignore this error and try to upload or import a demo
app, I get the same error.

What's odd is that the gateway does write into the HDFS
directory:

Found 6 items
drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
/user/dtadmin/datatorrent/appPackages
drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
/user/dtadmin/datatorrent/apps
drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
/user/dtadmin/datatorrent/conf
drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
/user/dtadmin/datatorrent/dashboards
drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
/user/dtadmin/datatorrent/licenses
drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
/user/dtadmin/datatorrent/systemAlerts

I've tried playing with different ways to configure the DFS
location, but nothing works.
If I use an actual local filesystem directory, I get a
different errro from the installation wizard (Non-DFS file system
is used: org.apache.hadoop.fs.LocalFileSystem), which makes sense.
But if I ignore this error, the app wizard successsfully writes
packages into the local directory, but I can't launch them (no
surprise there).

Anybody have any ideas what I can do?

thanks much.
-david

Reply via email to