Questions about Tez

Hello all, I've just started looking at Tez. I've setup Tez locally and have run sample MapReduce job with Tez as a replacement for Yarn MRV2. I plan to use Tez independently (No hive or pig).  I have the following questions 1.  Are there performance gains to use Tez for Mapreduce jobs apart

Re: Zip Exception since commit da4098b9

s like a Inflater-Deflater mismatch to me. From stack traces I see you cherry-picked this patch instead of using master branch.Would you mind double check whether the patch is correctly cherry-picked? Thanks!Zhiyuan On Sep 23, 2016, at 10:21 AM, Madhusudan Ramanna <m.rama...@ymail.com>

Re: Zip Exception since commit da4098b9

t 4:08 PM, Madhusudan Ramanna <m.rama...@ymail.com> wrote: > > Zhiyuan, > > We just pulled down the latest snapshot from Apache repository.  Question, is > how can I figure out branch and commit information from the snapshot artifact > ? > > thanks, > Madhu >

Re: Debugging M/R job with tez

Not sure if this will help, but If you're running into class loading issues , take a look here:  FAQ - Tez - Apache Software Foundation | | | | || | | | | | FAQ - Tez - Apache Software Foundation | | | | On Wednesday, September 28, 2016 9:39 AM,

Re: adding local resource to classpath and/or java.library.path

the AM and the task containers is to use “tez.cluster.additional.classpath.prefix” by setting it to something like “./archive name/*” — Hitesh > On Oct 4, 2016, at 4:38 PM, Madhusudan Ramanna <m.rama...@ymail.com> wrote: > > Actually, we solved it using > > tez.task.

adding local resource to classpath and/or java.library.path

Hello Folks, We have an archive local resource that is being expanded to  PWD//*.* How do we add PWD/ to classpath and java.library.path ? thanks,Madhu

Re: adding local resource to classpath and/or java.library.path

Actually, we solved it using tez.task.launch.env  in tez-site.xml thanks,Madhu On Tuesday, October 4, 2016 11:13 AM, Madhusudan Ramanna <m.rama...@ymail.com> wrote: Please note that we need this in the tez containers thanks,Madhu On Tuesday, October 4, 2016 11:11 AM, Madh

Zip Exception since commit da4098b9

Hello, We're using the Apache snapshot repository to pull latest tez snapshots.  We've started seeing this exception: org.apache.tez.dag.api.TezUncheckedException: java.util.zip.ZipException: incorrect header check at

Re: Node unable to start vertex

u hit any permission issues, feel free to reply back with your > confluence id. > > thanks > — Hitesh > > >> On Aug 25, 2016, at 12:59 PM, Madhusudan Ramanna <m.rama...@ymail.com> wrote: >> >> Thanks, #2 worked ! >> >> Might be a good id

Re: Node unable to start vertex

nd all tasks.  thanks — Hitesh > On Aug 25, 2016, at 11:46 AM, Madhusudan Ramanna <m.rama...@ymail.com> wrote: > > Hello, > > I'm trying to extend TezExamplesBase and get a dag running on yarn (pseudo > cluster mode on my host). > > For some reason, I'm ru

Re: question about yarn to tez webUI redirect

Likely you're using CDH yarn that is not patched yet.   I worked around this bug by adding a rewrite to tomcat RewriteRule ^/tez-ui/tez-ui(.*) /tez-ui$1 [R,NE,QSA] On Tuesday, September 27, 2016 11:16 AM, Ming Ma wrote: Hi, In our set up, the Tez UI is hosted under

Re: Tez containers and input splits

l+task+parallelism+works ). Each grouped split will be mapped to one task. This will then define what kind of container is requested. Let us know if you have more questions. thanks — Hitesh > On Oct 27, 2016, at 5:06 PM, Madhusudan Ramanna <m.rama...@ymail.com> wrote: > > Hell

Tez Sessions

Hello Folks, http://hortonworks.com/blog/introducing-tez-sessions/ >From the above post it seems like DAGs can only be executed serially.  Could >DAGs be executed in parallel on one Tez AM ?   thanks,Madhu

Container settings at vertex level

Hello Folks, Some vertices require more memory than other vertices. These vertices are memory intensive.  The graph, in general, takes a long(ish) time to complete.   Default allocation of a huge chunk of memory to this one DAG/application severely limits concurrent yarn containers that can be

Re: Tez Sessions

ld you provide more details on what you are trying to achieve? We could try and provide different viewpoints on trying to get you to a viable solution. — Hitesh > On Oct 20, 2016, at 10:52 AM, Madhusudan Ramanna <m.rama...@ymail.com> wrote: > > Hello Folks, > > http://horto

Vertex Parallelism

Hello Tez team, We have a native Tez application.  The first vertex in the graph is a downloader.  This vertex takes a CSV or sequence file that contains the "urls" as input, downloads content and passes the content on to the next vertex.  This input to vertex is smaller than the min split

Tez UI unable to display DAG

Hello Tez folks, Tez UI (0.8.4) is able to communicate with RM and display application ids and completion status.  It is however not displaying DAG or config information. http://host/ws/v1/cluster/apps/application_1481155336218_0289?_=1481308956420 (RM API) returns{ "app": { "id":

Re: Tez UI unable to display DAG

Harish, Here is what I'm seeing:  http://host:8188/ws/v1/timeline/TEZ_APPLICATION/tez_application_1481839368505_0008?_=1481852535448 returns - domain:"DEFAULT" - entity:"tez_application_1481839368505_0008" - entitytype:"TEZ_APPLICATION" - events:[] - otherinfo:{}

Question about yarn.timeline-service.enabled

Hello folks, I'm at a loss to explain why Tez thinks that yarn.timeline-service.enabled is false and therefore is not logging to timeline server.  I've search the classpath of the AM, located yarn-site.xml and made sure that  yarn.timeline-service.enabled is indeed set.  Any clues about what

Re: Question about yarn.timeline-service.enabled

We're in the process of setting up Tez UI, and nothing is being logged to timeline server and we see "no records available" in tez ui We already have the following config in tez-site.xml       Enable Tez to use the Timeline Server for History Logging    tez.history.logging.service.class