So do I !
I use PIG 0.6.x when I'm in development mode, then do some testings
in 0.8.0 before going to production.
PIG 0.6.0 local mode is SO fast that I cannot get rid of it.
Le 30/09/10 22:40, Konstantin Ignatyev a écrit :
Thanks.
I however wold rise my hands and feet to vote in support of faster
turnaround during development provided that execution path is not materially
different from production path.
The slowness of simple edit-try cycle is soo last century :)
Cheers!
On Thu, Sep 30, 2010 at 1:34 PM, Dmitriy Ryaboy<[email protected]> wrote:
Konstantin,
We explicitly chose to take out the thing that made Pig fast (faster,
anyway) in local mode because it exercised a different code path than
Hadoop-mode execution, which led to odd bugs and inconsistencies.
That being said, I think that with some work we could make local run a bit
faster, by pre-initializing the local hadoop threads and recycling them
between executions. This would be future work, though.. right now there is
no alternative.
You could of course downgrade to Pig 0.6, which is the last version to have
the fast local mode implementation. But then you have to watch out for the
aforementioned issues with unexpected differences vs hadoop mode.
-Dmitriy
On Thu, Sep 30, 2010 at 1:08 PM, Konstantin Ignatyev
<[email protected]>wrote:
Hi,
I am trying to write pig script that is quite complex so I am testing it
against very small data subset in local mode.
However it might take up to 2 _minutes_ to finish. Or 30 seconds if I
execute only parts of it.
That is quite annoying to say the least because SQL that I am trying to
reimplement in pig works on the source dataset for 3 seconds only.
Is there a way to improve PIG's speed in the local/development mode?
Thanks
--
Konstantin Ignatyev
PS: If this is a typical day on planet earth, humans will add fifteen
million tons of carbon to the atmosphere, destroy 115 square miles of
tropical rainforest, create seventy-two miles of desert, eliminate
between
forty to one hundred species, erode seventy-one million tons of topsoil,
add
2,700 tons of CFCs to the stratosphere, and increase their population by
263,000
Bowers, C.A. The Culture of Denial: Why the Environmental Movement Needs
a
Strategy for Reforming Universities and Public Schools. New York: State
University of New York Press, 1997: (4) (5) (p.206)
--
*Vincent BARAT, UBIKOD, CTO*
[email protected] <mailto:[email protected]> Mob +33 (0)6 15 41 15 18
UBIKOD Paris, c/o ESSEC VENTURES, Avenue Bernard Hirsch, 95021
Cergy-Pontoise cedex, FRANCE, Tel +33 (0)1 34 43 28 89
UBIKOD Rennes, 10 rue Duhamel, 35000 Rennes, FRANCE, Tel. +33 (0)2
99 65 69 13
www.ubikod.com <http://www.ubikod.com/>@ubikod
<http://twitter.com/ubikod>
www.capptain.com <http://www.capptain.com/>@capptain_hq
<http://twitter.com/capptain_hq>
IMPORTANT NOTICE -- UBIKOD and CAPPTAIN are registered trademarks of
UBIKOD S.A.R.L., all copyrights are reserved. The contents of this
email and attachments are confidential and may be subject to legal
privilege and/or protected by copyright. Copying or communicating
any part of it to others is prohibited and may be unlawful. If you
are not the intended recipient you must not use, copy, distribute or
rely on this email and should please return it immediately or notify
us by telephone. At present the integrity of email across the
Internet cannot be guaranteed. Therefore UBIKOD S.A.R.L. will not
accept liability for any claims arising as a result of the use of
this medium for transmissions by or to UBIKOD S.A.R.L.. UBIKOD
S.A.R.L. may exercise any of its rights under relevant law, to
monitor the content of all electronic communications. You should
therefore be aware that this communication and any responses might
have been monitored, and may be accessed by UBIKOD S.A.R.L. The
views expressed in this document are that of the individual and may
not necessarily constitute or imply its endorsement or
recommendation by UBIKOD S.A.R.L. The content of this electronic
mail may be subject to the confidentiality terms of a
"Non-Disclosure Agreement" (NDA).
