Re: [malhar-users] Not able to view applications in Datatorrent Monitor Tab

[David Yan]

Hi Sai:
Can you monitor the dtgateway.log file and send us the possible stack trace
that is causing the error?

David

On Mon, May 2, 2016 at 5:28 PM, Amol Kekre  wrote:

>
> Sai,
> I am redirecting this thread to users@apex. Do subscribe to users@ email.
> See http://apex.apache.org/community.html for details
>
> Thks,
> Amol
>
>
> On Mon, May 2, 2016 at 5:25 PM, Sai Karthik 
> wrote:
>
>> After launching the pi-demo or any demo application successfully, when I
>> click on the application id to monitor it, I get the error "An error
>> occurred fetching data.". Basically, I am not able to see any applications
>> in Monitor tab.
>> I can view these running or finished applications in Resource Manager UI.
>> Am I missing any configuration?
>>
>> My cluster details are
>> HDP 2.3.4
>> Java 1.8.0
>> DataTorrent Enterprise edition 30 day trial
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Malhar" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to malhar-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to malhar-us...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/malhar-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Malhar" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to malhar-users+unsubscr...@googlegroups.com.
> To post to this group, send email to malhar-us...@googlegroups.com.
> Visit this group at https://groups.google.com/group/malhar-users.
> For more options, visit https://groups.google.com/d/optout.
>

Re: Set schemaRequired differently on different operator implementations

[Thomas Weise]

Yes, you can override the port with a new port field with the same name.


On Mon, May 2, 2016 at 4:22 PM, Bhupesh Chawda 
wrote:

> Hi All,
>
> I have a base operator which is parametrized:
>
>
>
> *class OperatorBase extends BaseOperator{*
>
> *public final transient DefaultOutputPort output = new
> DefaultOutputPort();}*
>
> I also have two implementations of this operator as follows:
>
>- Instantiating with  type -
>- *class Operator1 extends OperatorBase*
>- POJO Implementation using  -
>- *class Operator1 extends OperatorBase*
>
> The problem is the following:
>
> I need to enable the annotation "*schemaRequired*" on the output port "
> *output*" only if a POJO implementation is used and not otherwise. Please
> suggest some way to achieve this.
>
> Can this be done without moving the port to individual child classes?
>
> Thanks.
>
> ~Bhupesh
>

Re: javax.persistence.Persistence classNotFoundError

[Ananth Gundabattula]

Hello Siyuan,

Please find attached the output of the unzip -t command on the apa package.

The library jar is indeed present in there.

What confuses me here is that the stack trace suggests that it is coming
when the engine is validating the App ( and not from my application code).
I am suspecting one of the following is causing this :


   - the bval ( JSR 303 ) jar validation path  is somehow conflicting with
   the JPA validation framework and is causing this issue but am not sure.
   - CDH 5.7 might have changed restructured a few jars ?? ( I am not sure
   of this either as I am not sure how Apex uses the underlying Hadoop
   distribution jars )

Pasting the stack trace once more here for reference:

An error occurred trying to launch the application. Server message:
java.lang.NoClassDefFoundError: javax/persistence/Persistence at
org.apache.bval.jsr303.resolver.JPATraversableResolver.isReachable(JPATraversableResolver.java:34)
at
org.apache.bval.jsr303.resolver.DefaultTraversableResolver.isReachable(DefaultTraversableResolver.java:60)
at
org.apache.bval.jsr303.resolver.CachingTraversableResolver.isReachable(CachingTraversableResolver.java:82)
at
org.apache.bval.jsr303.ConstraintValidation.isReachable(ConstraintValidation.java:241)
at
org.apache.bval.jsr303.ConstraintValidation.validate(ConstraintValidation.java:166)
at
org.apache.bval.jsr303.ConstraintValidation.validate(ConstraintValidation.java:141)
at
org.apache.bval.util.ValidationHelper.validateProperty(ValidationHelper.java:233)
at
org.apache.bval.util.ValidationHelper.validateBean(ValidationHelper.java:216)
at
org.apache.bval.jsr303.ClassValidator.validateBeanNet(ClassValidator.java:393)
at org.apache.bval.jsr303.ClassValidator.validate(ClassValidator.java:149)
at
com.datatorrent.stram.plan.logical.LogicalPlan.validate(LogicalPlan.java:1672)
at com.datatorrent.stram.StramClient.(StramClient.java:161) at
com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:509)
at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2050)
at com.datatorrent.stram.cli.DTCli.launchAppPackage(DTCli.java:3456) at
com.datatorrent.stram.cli.DTCli.access$7100(DTCli.java:106) at
com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:1895) at
com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1449) Fatal error
encountered




Regards,
Ananth

On Tue, May 3, 2016 at 3:22 AM, hsy...@gmail.com  wrote:

> Hey Ananth,
>
> Can you unzip your apa file to see if it is in the lib folder? If it is
> not there, it means the packaging is not correct.
>
> Regards,
> Siyuan
>
> On Mon, May 2, 2016 at 4:39 AM, Ananth Gundabattula <
> agundabatt...@gmail.com> wrote:
>
>> Hello All,
>>
>> I was able to get around the issue pasted above by manually copying the
>> javaee-api-7.0 jar inside the lib folder of Apex engine. It may be noted
>> that the code works perfectly fine in a unit test that launches all of the
>> operators in the DAG. ( Works end to end ) . However when I launch it on
>> CDH 5.7 I run into the exception pasted above. It may be noted that the
>> exception is being raised when I launch the app and the Apex engine is
>> trying to validate the newly deployed app.
>>
>> I am suspecting that this could be an issue with incompatibility with CDH
>> 5.7 ( The Hadoop stack on which I am trying to run Apex 3.3 on )
>>
>> Could anyone help me if they were able to run Apex on top of CDH 5.7. I
>> got the same exception with both Apex 3.3 and 3.2.
>>
>> Regards,
>> Ananth
>>
>> On Sat, Apr 30, 2016 at 11:34 AM, Ananth Gundabattula <
>> agundabatt...@gmail.com> wrote:
>>
>>> Hello All,
>>>
>>> I am getting the following exception when I launch my Apex app.
>>>
>>> I tried including javaee-api- version 7.0 as well to get around this
>>> error but does not seem to take effect.
>>>
>>> Any ideas why javax.persistence.Persistence is not getting loaded by the
>>> APex engine classloader ?
>>>
>>> An error occurred trying to launch the application. Server message:
 java.lang.NoClassDefFoundError: javax/persistence/Persistence at
 org.apache.bval.jsr303.resolver.JPATraversableResolver.isReachable(JPATraversableResolver.java:34)
 at
 org.apache.bval.jsr303.resolver.DefaultTraversableResolver.isReachable(DefaultTraversableResolver.java:60)
 at
 org.apache.bval.jsr303.resolver.CachingTraversableResolver.isReachable(CachingTraversableResolver.java:82)
 at
 org.apache.bval.jsr303.ConstraintValidation.isReachable(ConstraintValidation.java:241)
 at
 org.apache.bval.jsr303.ConstraintValidation.validate(ConstraintValidation.java:166)
 at
 org.apache.bval.jsr303.ConstraintValidation.validate(ConstraintValidation.java:141)
 at
 org.apache.bval.util.ValidationHelper.validateProperty(ValidationHelper.java:233)
 at
 org.apache.bval.util.ValidationHelper.validateBean(ValidationHelper.java:216)
 at
 

Re: javax.persistence.Persistence classNotFoundError

[hsy...@gmail.com]

Hey Ananth,

Can you unzip your apa file to see if it is in the lib folder? If it is not
there, it means the packaging is not correct.

Regards,
Siyuan

On Mon, May 2, 2016 at 4:39 AM, Ananth Gundabattula  wrote:

> Hello All,
>
> I was able to get around the issue pasted above by manually copying the
> javaee-api-7.0 jar inside the lib folder of Apex engine. It may be noted
> that the code works perfectly fine in a unit test that launches all of the
> operators in the DAG. ( Works end to end ) . However when I launch it on
> CDH 5.7 I run into the exception pasted above. It may be noted that the
> exception is being raised when I launch the app and the Apex engine is
> trying to validate the newly deployed app.
>
> I am suspecting that this could be an issue with incompatibility with CDH
> 5.7 ( The Hadoop stack on which I am trying to run Apex 3.3 on )
>
> Could anyone help me if they were able to run Apex on top of CDH 5.7. I
> got the same exception with both Apex 3.3 and 3.2.
>
> Regards,
> Ananth
>
> On Sat, Apr 30, 2016 at 11:34 AM, Ananth Gundabattula <
> agundabatt...@gmail.com> wrote:
>
>> Hello All,
>>
>> I am getting the following exception when I launch my Apex app.
>>
>> I tried including javaee-api- version 7.0 as well to get around this
>> error but does not seem to take effect.
>>
>> Any ideas why javax.persistence.Persistence is not getting loaded by the
>> APex engine classloader ?
>>
>> An error occurred trying to launch the application. Server message:
>>> java.lang.NoClassDefFoundError: javax/persistence/Persistence at
>>> org.apache.bval.jsr303.resolver.JPATraversableResolver.isReachable(JPATraversableResolver.java:34)
>>> at
>>> org.apache.bval.jsr303.resolver.DefaultTraversableResolver.isReachable(DefaultTraversableResolver.java:60)
>>> at
>>> org.apache.bval.jsr303.resolver.CachingTraversableResolver.isReachable(CachingTraversableResolver.java:82)
>>> at
>>> org.apache.bval.jsr303.ConstraintValidation.isReachable(ConstraintValidation.java:241)
>>> at
>>> org.apache.bval.jsr303.ConstraintValidation.validate(ConstraintValidation.java:166)
>>> at
>>> org.apache.bval.jsr303.ConstraintValidation.validate(ConstraintValidation.java:141)
>>> at
>>> org.apache.bval.util.ValidationHelper.validateProperty(ValidationHelper.java:233)
>>> at
>>> org.apache.bval.util.ValidationHelper.validateBean(ValidationHelper.java:216)
>>> at
>>> org.apache.bval.jsr303.ClassValidator.validateBeanNet(ClassValidator.java:393)
>>> at org.apache.bval.jsr303.ClassValidator.validate(ClassValidator.java:149)
>>> at
>>> com.datatorrent.stram.plan.logical.LogicalPlan.validate(LogicalPlan.java:1672)
>>> at com.datatorrent.stram.StramClient.(StramClient.java:161) at
>>> com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:509)
>>> at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2050)
>>> at com.datatorrent.stram.cli.DTCli.launchAppPackage(DTCli.java:3456) at
>>> com.datatorrent.stram.cli.DTCli.access$7100(DTCli.java:106) at
>>> com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:1895) at
>>> com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1449) Fatal error
>>> encountered
>>
>>
>> Regards,
>> Ananth
>>
>
>

Re: javax.persistence.Persistence classNotFoundError

[Ananth Gundabattula]

Hello All,

I was able to get around the issue pasted above by manually copying the
javaee-api-7.0 jar inside the lib folder of Apex engine. It may be noted
that the code works perfectly fine in a unit test that launches all of the
operators in the DAG. ( Works end to end ) . However when I launch it on
CDH 5.7 I run into the exception pasted above. It may be noted that the
exception is being raised when I launch the app and the Apex engine is
trying to validate the newly deployed app.

I am suspecting that this could be an issue with incompatibility with CDH
5.7 ( The Hadoop stack on which I am trying to run Apex 3.3 on )

Could anyone help me if they were able to run Apex on top of CDH 5.7. I got
the same exception with both Apex 3.3 and 3.2.

Regards,
Ananth

On Sat, Apr 30, 2016 at 11:34 AM, Ananth Gundabattula <
agundabatt...@gmail.com> wrote:

> Hello All,
>
> I am getting the following exception when I launch my Apex app.
>
> I tried including javaee-api- version 7.0 as well to get around this error
> but does not seem to take effect.
>
> Any ideas why javax.persistence.Persistence is not getting loaded by the
> APex engine classloader ?
>
> An error occurred trying to launch the application. Server message:
>> java.lang.NoClassDefFoundError: javax/persistence/Persistence at
>> org.apache.bval.jsr303.resolver.JPATraversableResolver.isReachable(JPATraversableResolver.java:34)
>> at
>> org.apache.bval.jsr303.resolver.DefaultTraversableResolver.isReachable(DefaultTraversableResolver.java:60)
>> at
>> org.apache.bval.jsr303.resolver.CachingTraversableResolver.isReachable(CachingTraversableResolver.java:82)
>> at
>> org.apache.bval.jsr303.ConstraintValidation.isReachable(ConstraintValidation.java:241)
>> at
>> org.apache.bval.jsr303.ConstraintValidation.validate(ConstraintValidation.java:166)
>> at
>> org.apache.bval.jsr303.ConstraintValidation.validate(ConstraintValidation.java:141)
>> at
>> org.apache.bval.util.ValidationHelper.validateProperty(ValidationHelper.java:233)
>> at
>> org.apache.bval.util.ValidationHelper.validateBean(ValidationHelper.java:216)
>> at
>> org.apache.bval.jsr303.ClassValidator.validateBeanNet(ClassValidator.java:393)
>> at org.apache.bval.jsr303.ClassValidator.validate(ClassValidator.java:149)
>> at
>> com.datatorrent.stram.plan.logical.LogicalPlan.validate(LogicalPlan.java:1672)
>> at com.datatorrent.stram.StramClient.(StramClient.java:161) at
>> com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:509)
>> at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2050)
>> at com.datatorrent.stram.cli.DTCli.launchAppPackage(DTCli.java:3456) at
>> com.datatorrent.stram.cli.DTCli.access$7100(DTCli.java:106) at
>> com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:1895) at
>> com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1449) Fatal error
>> encountered
>
>
> Regards,
> Ananth
>

WebService errors in DT 3.3/3.2

[Ananth Gundabattula]

Hello All,

I am trying to launch my app on Apex/DT 3.3 version running on top of CDH
5.7. I tried on DT/Apex 3.2 as well running on CDH 5.7. Both these
deployments show the following behavior. (We have a requirement to run Apex
on CDH 5.7 )

I was wondering if you could advise on any workarounds for the following
issue:

I am not able to see any metrics in the web console when I open the
"Monitor" section. The application code seems to be working fine as I see
data flow happening fine as expected. However I see no metrics ( even the
top level container metrics) .

One more observed issue is that when I click on the container ID from the
monitor section, the UI crashes. ( In the sense the UI simply does not show
anything ). I am however able to get some preliminary UI ( barring the
metrics ) if I click on the container link from the monitor section if the
application is in the "ACCEPTED" state. The container link does not work
the moment the app moves from ACCEPTED to Running State.


The DtGateway shows the following logs :

2016-05-02 11:08:48,964 WARN com.datatorrent.stram.client.StramAgent:
> Caught exception when retrieving web service info for app
> application_1462007878015_0037
> org.codehaus.jettison.json.JSONException: A JSONObject text must begin
> with '{' at character 1 of 
> 
> 
> Error 500
> javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
> 
> HTTP ERROR 500
> Problem accessing /ws. Reason:
> 
> javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;Caused
> by:java.lang.AbstractMethodError:
> javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
> at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:119)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:870)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1286)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> 
> Powered by Jetty://
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
> 
>
> at
> org.codehaus.jettison.json.JSONTokener.syntaxError(JSONTokener.java:439)
> at org.codehaus.jettison.json.JSONObject.(JSONObject.java:169)
> at org.codehaus.jettison.json.JSONObject.(JSONObject.java:266)
> at
> com.datatorrent.stram.client.StramAgent.retrieveWebServicesInfo(StramAgent.java:402)
> at
>