Re: tez and beeline and hs2

[Hitesh Shah]

As Gopal mentioned, it is optional. The log message could probably be set to a 
WARN to not confuse users. 

— Hitesh 

On Feb 25, 2016, at 7:26 AM, Stephen Sprague <sprag...@gmail.com> wrote:

> hey guys,
> still not getting jobs to the running state via tez->beeline->hs2.
> 
> lemme ask this first:  Is is mandatory that the Tez UI be up an running for 
> this to work?  
> 
> this looks to be a hard error (see below) but given the "local cli" works i 
> don't think its *mandatory*. But maybe via HS2 it is?
> 
> from yarn logs:
> 2016-02-25 07:07:56,744 [ERROR] [main] |web.WebUIService|: Tez UI History URL 
> is not set
> 
> Thanks,
> Stephen
> PS setting up the web service is just another step and complication i'm 
> hoping to avoid for just testing. :)
> 
> On Mon, Feb 22, 2016 at 10:43 AM, Bikas Saha <bi...@apache.org> wrote:
> Hi Stephen,
> 
>  
> 
> Thanks for bearing with any delays on our side and keeping us updated. In the 
> end if we figure out that this is less about bugs in Tez/Hive and more about 
> lack of well documented best practices, then it would be useful to produce a 
> wiki page about this.
> 
>  
> 
> Thanks
> 
> Bikas
> 
>  
> 
> From: Stephen Sprague [mailto:sprag...@gmail.com] 
> Sent: Monday, February 22, 2016 6:59 AM
> To: user@tez.apache.org
> Subject: Re: tez and beeline and hs2
> 
>  
> 
> just an update.  i haven't given up!  i've just been pulled into other things 
> this weekend and am hoping to pick it up again this week.
> 
>  
> 
> On Fri, Feb 19, 2016 at 9:30 AM, Hitesh Shah <hit...@apache.org> wrote:
> 
> Not exactly. I think the UI bits might be a red-herring. Bouncing YARN and 
> HS2 also should unlikely be needed unless you are modifying configs.
> 
> There is likely a bug ( the NPE being logged ) in the shutting down code for  
> the org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService ( if 
> it was not started properly ) but the fact that it is shutting down means 
> that there is something else wrong ( which should have happened before the 
> shutdown sequence kicked in ). Feel free to file a bug with the logs attached 
> if you cannot attach them over the mailing list here.
> 
> thanks
> — Hitesh
> 
> 
> 
> On Feb 19, 2016, at 7:16 AM, Stephen Sprague <sprag...@gmail.com> wrote:
> 
> > Hi Gopal,
> > nice. I should have known that command as you and Hitesh have given that 
> > advice in the past on various threads here! sorry for that.
> >
> > And sure enough the smoking gun reveals itself.  thank you.
> >
> > {quote}
> > 2016-02-18 18:40:19,672 [ERROR] [main] |web.WebUIService|: Tez UI History 
> > URL is not set
> > 2016-02-18 18:40:19,731 [WARN] 
> > [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] 
> > |service.AbstractService|: When stopping the service 
> > org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService : 
> > java.lang.NullPointerException
> > java.lang.NullPointerException
> > {quote}
> >
> > So it looks like this Tez UI app is required for 'jdbc' mode.  Lemme 
> > research that puppy and perhaps bounce Yarn and HS2.
> >
> > Thanks again for shining the light!
> >
> > Cheeers,
> > Stephen.
> >
> > On Thu, Feb 18, 2016 at 10:09 PM, Gopal Vijayaraghavan <gop...@apache.org> 
> > wrote:
> >
> > >
> > >http://dwrdevnn1.sv2.trulia.com:8088/proxy/application_1455811467110_0307/
> > >
> > ><http://dwrdevnn1.sv2.trulia.com:8088/proxy/application_1455811467110_0307
> > >/>
> > ...
> > > So my question is whatya suppose is causing this?  I'm pretty darn sure
> > >the classpath is legit now.
> >
> > Two steps forward, one step back :)
> >
> > yarn logs -applicationId application_1455811467110_0307 for answers.
> >
> > Cheers,
> > Gopal
> >
> >
> >
> 
>  
> 
> 

Re: tez and beeline and hs2

[Stephen Sprague]

Awesome. Thank you Gopal.

I re-copied yarn-site.xml and tez-site.xml to all nodes in the cluster and
i restarted yarn - but dang it all i forgot to restart HS2.  Once i did
that this morning after reviewing my steps we now have lift-off!

Naturally it was fumbling on my part and the hadoop classpath that really
tripped me up but i think we're pretty good now.

You guys have been totally responsive here and it has been much
appreciated.

Thanks again!

Cheers,
Stephen.

On Thu, Feb 25, 2016 at 7:32 AM, Gopal Vijayaraghavan 
wrote:

> > lemme ask this first:  Is is mandatory that the Tez UI be up an running
> >for this to work?
>
> Nope. The UI is entirely optional.
>
> Although you might want to be double-sure and disable the UI's data writer
> too.
>
> tez.history.logging.service.class
> org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService<
> /value>
>
>
> Cheers,
> Gopal
>
>
>

Re: tez and beeline and hs2

[Gopal Vijayaraghavan]

> lemme ask this first:  Is is mandatory that the Tez UI be up an running
>for this to work? 

Nope. The UI is entirely optional.

Although you might want to be double-sure and disable the UI's data writer
too.

tez.history.logging.service.class
org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService<
/value>


Cheers,
Gopal

Re: tez and beeline and hs2

[Stephen Sprague]

hey guys,
still not getting jobs to the running state via tez->beeline->hs2.

lemme ask this first:  Is is mandatory that the Tez UI be up an running for
this to work?

this looks to be a hard error (see below) but given the "local cli" works i
don't think its *mandatory*. But maybe via HS2 it is?

from yarn logs:
2016-02-25 07:07:56,744 [ERROR] [main] |web.WebUIService|: Tez UI History
URL is not set

Thanks,
Stephen
PS setting up the web service is just another step and complication i'm
hoping to avoid for just testing. :)

On Mon, Feb 22, 2016 at 10:43 AM, Bikas Saha <bi...@apache.org> wrote:

> Hi Stephen,
>
>
>
> Thanks for bearing with any delays on our side and keeping us updated. In
> the end if we figure out that this is less about bugs in Tez/Hive and more
> about lack of well documented best practices, then it would be useful to
> produce a wiki page about this.
>
>
>
> Thanks
>
> Bikas
>
>
>
> *From:* Stephen Sprague [mailto:sprag...@gmail.com]
> *Sent:* Monday, February 22, 2016 6:59 AM
> *To:* user@tez.apache.org
> *Subject:* Re: tez and beeline and hs2
>
>
>
> just an update.  i haven't given up!  i've just been pulled into other
> things this weekend and am hoping to pick it up again this week.
>
>
>
> On Fri, Feb 19, 2016 at 9:30 AM, Hitesh Shah <hit...@apache.org> wrote:
>
> Not exactly. I think the UI bits might be a red-herring. Bouncing YARN and
> HS2 also should unlikely be needed unless you are modifying configs.
>
> There is likely a bug ( the NPE being logged ) in the shutting down code
> for  the
> org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService ( if it
> was not started properly ) but the fact that it is shutting down means that
> there is something else wrong ( which should have happened before the
> shutdown sequence kicked in ). Feel free to file a bug with the logs
> attached if you cannot attach them over the mailing list here.
>
> thanks
> — Hitesh
>
>
>
> On Feb 19, 2016, at 7:16 AM, Stephen Sprague <sprag...@gmail.com> wrote:
>
> > Hi Gopal,
> > nice. I should have known that command as you and Hitesh have given that
> advice in the past on various threads here! sorry for that.
> >
> > And sure enough the smoking gun reveals itself.  thank you.
> >
> > {quote}
> > 2016-02-18 18:40:19,672 [ERROR] [main] |web.WebUIService|: Tez UI
> History URL is not set
> > 2016-02-18 18:40:19,731 [WARN]
> [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler]
> |service.AbstractService|: When stopping the service
> org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService :
> java.lang.NullPointerException
> > java.lang.NullPointerException
> > {quote}
> >
> > So it looks like this Tez UI app is required for 'jdbc' mode.  Lemme
> research that puppy and perhaps bounce Yarn and HS2.
> >
> > Thanks again for shining the light!
> >
> > Cheeers,
> > Stephen.
> >
> > On Thu, Feb 18, 2016 at 10:09 PM, Gopal Vijayaraghavan <
> gop...@apache.org> wrote:
> >
> > >
> > >
> http://dwrdevnn1.sv2.trulia.com:8088/proxy/application_1455811467110_0307/
> > >
> > ><
> http://dwrdevnn1.sv2.trulia.com:8088/proxy/application_1455811467110_0307
> > >/>
> > ...
> > > So my question is whatya suppose is causing this?  I'm pretty darn sure
> > >the classpath is legit now.
> >
> > Two steps forward, one step back :)
> >
> > yarn logs -applicationId application_1455811467110_0307 for answers.
> >
> > Cheers,
> > Gopal
> >
> >
> >
>
>
>

Re: tez and beeline and hs2

[Stephen Sprague]

just an update.  i haven't given up!  i've just been pulled into other
things this weekend and am hoping to pick it up again this week.

On Fri, Feb 19, 2016 at 9:30 AM, Hitesh Shah  wrote:

> Not exactly. I think the UI bits might be a red-herring. Bouncing YARN and
> HS2 also should unlikely be needed unless you are modifying configs.
>
> There is likely a bug ( the NPE being logged ) in the shutting down code
> for  the
> org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService ( if it
> was not started properly ) but the fact that it is shutting down means that
> there is something else wrong ( which should have happened before the
> shutdown sequence kicked in ). Feel free to file a bug with the logs
> attached if you cannot attach them over the mailing list here.
>
> thanks
> — Hitesh
>
>
> On Feb 19, 2016, at 7:16 AM, Stephen Sprague  wrote:
>
> > Hi Gopal,
> > nice. I should have known that command as you and Hitesh have given that
> advice in the past on various threads here! sorry for that.
> >
> > And sure enough the smoking gun reveals itself.  thank you.
> >
> > {quote}
> > 2016-02-18 18:40:19,672 [ERROR] [main] |web.WebUIService|: Tez UI
> History URL is not set
> > 2016-02-18 18:40:19,731 [WARN]
> [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler]
> |service.AbstractService|: When stopping the service
> org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService :
> java.lang.NullPointerException
> > java.lang.NullPointerException
> > {quote}
> >
> > So it looks like this Tez UI app is required for 'jdbc' mode.  Lemme
> research that puppy and perhaps bounce Yarn and HS2.
> >
> > Thanks again for shining the light!
> >
> > Cheeers,
> > Stephen.
> >
> > On Thu, Feb 18, 2016 at 10:09 PM, Gopal Vijayaraghavan <
> gop...@apache.org> wrote:
> >
> > >
> > >
> http://dwrdevnn1.sv2.trulia.com:8088/proxy/application_1455811467110_0307/
> > >
> > ><
> http://dwrdevnn1.sv2.trulia.com:8088/proxy/application_1455811467110_0307
> > >/>
> > ...
> > > So my question is whatya suppose is causing this?  I'm pretty darn sure
> > >the classpath is legit now.
> >
> > Two steps forward, one step back :)
> >
> > yarn logs -applicationId application_1455811467110_0307 for answers.
> >
> > Cheers,
> > Gopal
> >
> >
> >
>
>

Re: tez and beeline and hs2

[Gopal Vijayaraghavan]

> 
>http://dwrdevnn1.sv2.trulia.com:8088/proxy/application_1455811467110_0307/
> 
>/>
...
> So my question is whatya suppose is causing this?  I'm pretty darn sure
>the classpath is legit now.

Two steps forward, one step back :)

yarn logs -applicationId application_1455811467110_0307 for answers.

Cheers,
Gopal

tez and beeline and hs2

[Stephen Sprague]

Hey guys,
I back on the tez user group just because i think my problem is more
focused on Tez -- but lemme know if you think otherwise.

Thanks to Gopal again for getting me to think about the hadoop classpath
when starting HS2. I again changed classpath in the same way i had to
change it for the hive client. So now i've got much further down the Tez
path via HS2 but not quite all the way.

I get to the stage where the Tez job is ACCEPTED by yarn but there it
encounters some communication issues trying to communicate with the
Application Manager on the data nodes (i think)

Here's a snippet from the HS2 log now.

{code}
2016-02-18 18:40:12,259 INFO  [HiveServer2-Background-Pool: Thread-157]:
tez.TezSessionState (TezSessionState.java:open(180)) - Opening new Tez
Session (id: 5d94006e-8a85-4ed0-a8ce-e9c86d1f3d28, scratch dir: hdfs://
dwrdevnn1.sv2.trulia.com:8020/tmp/hive/dwr/_tez_session_dir/5d94006e-8a85-4ed0-a8ce-e9c86d1f3d28
)
2016-02-18 18:40:12,292 INFO  [HiveServer2-Background-Pool: Thread-157]:
client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to
ResourceManager at dwrdevnn1.sv2.trulia.com/172.19.103.136:8032
2016-02-18 18:40:12,293 INFO  [HiveServer2-Background-Pool: Thread-157]:
client.TezClient (TezClient.java:start(394)) - Session mode. Starting
session.
2016-02-18 18:40:12,294 INFO  [HiveServer2-Background-Pool: Thread-157]:
client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(176))
- Using tez.lib.uris value from configuration: hdfs://
dwrdevnn1.sv2.trulia.com:8020/apps/tez-0.8.2,hdfs://dwrdevnn1.sv2.trulia.com:8020/apps/tez-0.8.2/lib
2016-02-18 18:40:12,354 INFO  [HiveServer2-Background-Pool: Thread-157]:
client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) -
Tez system stage directory hdfs://
dwrdevnn1.sv2.trulia.com:8020/tmp/hive/dwr/_tez_session_dir/5d94006e-8a85-4ed0-a8ce-e9c86d1f3d28/.tez/application_1455811467110_0307
doesn't exist and is created
2016-02-18 18:40:12,360 INFO  [HiveServer2-Background-Pool: Thread-157]:
Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1087)) -
fs.default.name is deprecated. Instead, use fs.defaultFS
2016-02-18 18:40:12,525 INFO  [HiveServer2-Background-Pool: Thread-157]:
impl.YarnClientImpl (YarnClientImpl.java:submitApplication(251)) -
Submitted application application_1455811467110_0307
2016-02-18 18:40:12,528 INFO  [HiveServer2-Background-Pool: Thread-157]:
client.TezClient (TezClient.java:start(428)) - The url to track the Tez
Session:
http://dwrdevnn1.sv2.trulia.com:8088/proxy/application_1455811467110_0307/
2016-02-18 18:40:18,122 INFO  [HiveServer2-Background-Pool: Thread-157]:
ipc.Client (Client.java:handleConnectionFailure(858)) - Retrying connect to
server: dwrdevdn13.sv2.trulia.com/172.19.79.129:44618. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2016-02-18 18:40:19,123 INFO  [HiveServer2-Background-Pool: Thread-157]:
ipc.Client (Client.java:handleConnectionFailure(858)) - Retrying connect to
server: dwrdevdn13.sv2.trulia.com/172.19.79.129:44618. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2016-02-18 18:40:20,124 INFO  [HiveServer2-Background-Pool: Thread-157]:
ipc.Client (Client.java:handleConnectionFailure(858)) - Retrying connect to
server: dwrdevdn13.sv2.trulia.com/172.19.79.129:44618. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2016-02-18 18:40:21,125 INFO  [HiveServer2-Background-Pool: Thread-157]:
ipc.Client (Client.java:handleConnectionFailure(858)) - Retrying connect to
server: dwrdevdn13.sv2.trulia.com/172.19.79.129:44618. Already tried 3
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2016-02-18 18:40:22,126 INFO  [HiveServer2-Background-Pool: Thread-157]:
ipc.Client (Client.java:handleConnectionFailure(858)) - Retrying connect to
server: dwrdevdn13.sv2.trulia.com/172.19.79.129:44618. Already tried 4
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2016-02-18 18:40:23,128 INFO  [HiveServer2-Background-Pool: Thread-157]:
ipc.Client (Client.java:handleConnectionFailure(858)) - Retrying connect to
server: dwrdevdn13.sv2.trulia.com/172.19.79.129:44618. Already tried 5
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2016-02-18 18:40:24,129 INFO  [HiveServer2-Background-Pool: Thread-157]:
ipc.Client (Client.java:handleConnectionFailure(858)) - Retrying connect to
server: dwrdevdn13.sv2.trulia.com/172.19.79.129:44618. Already tried 6
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2016-02-18 18:40:25,130 INFO  [HiveServer2-Background-Pool: Thread-157]:
ipc.Client (Client.java:handleConnectionFailure(858)) - Retrying connect to