For the performance tuning of Tez, you can refer Rajesh’s reply before,
> On Fri, Apr 24, 2015 at 1:54 AM, Rajesh Balamohan <[email protected]> wrote: > Listing some details at very high level, > > - Set "tez.task.generate.counters.per.io=true" to get more details on the > task counters. Basically this starts printinng the counters per edge, which can be a lot more useful for debugging. > > - In case you want to avoid container launches etc when you analyze for first > time, try hive.prewarm.enabled=true & hive.prewarm.numcontainers=<no of containers you want in your sesssion to be prewarmed> > > - Container reuse is enabled by default in tez. > (tez.am.container.idle.release-timeout-min.millis, tez.am.container.idle.release-timeout-max.millis controls the amount of time a container is held by AM before releasing it) > > - Set tez.runtime.io.sort.mb appropriately to avoid spills (you can check > task counters in the logs to find out the spills and adjust it accordingly) > > - Set tez.runtime.sort.threads=2 to enable PipelinedSorter which is a lot > performant than DefaultSorter (this is the default in master branch. But if you are using earlier releases, you can turn it on by setting tez.runtime.sort.threads=2). > > - Set tez.runtime.compress=true and set tez.runtime.compress.codec > (SnappyCodec is preferred, but it is upto you to choose) > > - Set tez.runtime.shuffle.keep-alive.enabled=true in case you have shuffle > heavy workload. This reduces number of connections in shuffle. > > - Adjust memory allocated to different inputs/outputs based on > tez.task.scale.memory.ratios (but this is more of expert level setting which you might want to touch after nailing down any memory pressure) > > - Adjusting shuffle buffers are also possible, but would advise only when you > nail down an issue related to shuffle/merge codepath. > > - Set "tez.runtime.optimize.local.fetch=true" to bypass http fetches (when > data is locally Present) Best Regard, Jeff Zhang From: Mich Talebzadeh <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, December 31, 2015 at 9:28 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: RE: Getting error compiling TEZ 0.7 Thanks I am trying to use TEZ as execution engine for hive instead of MR. I have Hive 1.2.1 on Spark 1.3.1 working fine. That is the only version of Spark that works as execution engine for Hive (not the latest 1.5.1 I am afraid). The problem is that the performance is better than Hive on MR when I use Spark but otherwise the performance on joining 1 billion rows with another table takes 17 minutes compared to the same join in Sybase IQ DW that takes 30 seconds! The reason is attributed to Spark 1.3.1 not having full optimisation capabilities as it is older version So I would like to try Hive on TEZ which basically is MR on DAG. Hopefully that may give a better performance. Mich Talebzadeh Sybase ASE 15 Gold Medal Award 2008 A Winning Strategy: Running the most Critical Financial Data on ASE 15 http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7. co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4 Publications due shortly: Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8 Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/> NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Bikas Saha Sent: 31 December 2015 01:21 To: [email protected]<mailto:[email protected]> Subject: RE: Getting error compiling TEZ 0.7 Also, if 0.7.1 turns out to be a binary release (as currently planned) then you probably wont have to build Tez from scratch after that. From: Jianfeng (Jeff) Zhang [mailto:[email protected]] Sent: Wednesday, December 30, 2015 4:53 PM To: [email protected]<mailto:[email protected]> Subject: Re: Getting error compiling TEZ 0.7 This is a known issue which is fixed in 0.7.1, please try to use maven 3.3+ https://issues.apache.org/jira/browse/TEZ-2560 Best Regard, Jeff Zhang From: Mich Talebzadeh <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, December 31, 2015 at 4:54 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Getting error compiling TEZ 0.7 Hi, Trying to build TEZ 0.7 in /usr/lib/apache-tez-0.7.0-src using mvn -X clean package -DskipTests=true -Dmaven.javadoc.skip=true with mvn version 3.2.5 (as opposed to 3.3) as I read that I can build it OK with 3.2.5 following the same error ass below mvn --version Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T17:29:23+00:00) Maven home: /usr/local/apache-maven/apache-maven-3.2.5 Java version: 1.7.0_25, vendor: Oracle Corporation Java home: /usr/java/jdk1.7.0_25/jre I get this error [INFO] tez-ui ............................................. FAILURE [ 0.411 s] [ DEBUG] -- end configuration -- [INFO] Running 'npm install --color=false' in /usr/lib/apache-tez-0.7.0-src/tez-ui/src/main/webapp [INFO] /usr/lib/apache-tez-0.7.0-src/tez-ui/src/main/webapp/node/with_new_path.sh: line 3: 23781 Aborted "$@" [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.16:npm (npm install) on project tez-ui: Failed to run task: 'npm install --color=false' failed. (error code 134) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.16:npm (npm install) on project tez-ui: Failed to run task any ideas as there is little info available in net. Thanks Mich Talebzadeh Sybase ASE 15 Gold Medal Award 2008 A Winning Strategy: Running the most Critical Financial Data on ASE 15 http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7. co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4 Publications due shortly: Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8 Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/> NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
