Will the secondarysort api allow you to do anything for this, or will that only do ascending?
https://crunch.apache.org/user-guide.html#secsort https://crunch.apache.org/apidocs/0.10.0/org/apache/crunch/lib/SecondarySort.html On Wed, May 17, 2017 at 10:06 AM Giovanni GATTI PINHEIRO < [email protected]> wrote: > This is the first thing that crossed my mind, but it won’t work (at least > using Avro reflection). I think that the reason is that Crunch will not > deserialize the record to perform sorting and he will instead use a > built-in Avro function to do so. As a result, compareTo will have no > effect. This is pure guessing, I don’t have deep Avro-Crunch knowledge. > > > > What I did as work-around was to load all records in-memory and sort them > by myself. It works when you have a small set of records “per grouping > key”, but it won’t work with a large set (or with high memory constraints). > > > > So, I still quite curious if there is a “clean” solution to this or not. > > > > Thank you, > > > > *Giovanni GATTI PINHEIRO* > *Ingénieur Développement* > Volt Europe pour Médiamétrie > > > > www.mediametrie.fr > <https://mediametrie.letsignit.com/r/0/de904011-b6f5-434d-9596-03f0e4cbe9c0/ddffef52-9bfb-412f-80c2-ecc6903d7566> > > > > > > *De :* Dmitry Gorbatsevich [mailto: > [email protected]] > *Envoyé :* mercredi 17 mai 2017 15:11 > *À :* [email protected] > *Objet :* Re: Custom SecondarySort > > > > Hey, I have never tried this before, but just curious wether it is > possible to do by overriding compareTo method of your "target fields" > class. > > > > Thanks, > > Dmitry > > > > *From: *Giovanni GATTI PINHEIRO <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Wednesday, May 17, 2017 at 14:52 > *To: *"[email protected]" <[email protected]> > *Subject: *Custom SecondarySort > > > > Hello, > > > > I would like to know if there is an way to implement a custom secondary > sort, that is, instead of applying ascendant sorting on all fields target > fields, I would like to perform something else. If yes, how could that be > done? I couldn’t find how/if it could be done in Crunch documentation. > > > > Thank you in advance. > > > > Best Regards, > > > > *Giovanni GATTI PINHEIRO* > *Ingénieur Développement* > Volt Europe pour Médiamétrie > > > > Email : [email protected] > Buropolis Les Bouillides - 1240, Route des Dolines > 06560 Valbonne Sophia-Antipolis - France > > www.mediametrie.fr > <https://mediametrie.letsignit.com/r/0/5d5cef9f-1c0c-47fc-916f-f206f054bce8/462cef18-f2ef-483c-adc8-0c4d30473036> > > > > > <https://mediametrie.letsignit.com/r/5/801a3736-b2f8-4a60-a04a-a5669668bf74/829465c3-29f3-48c5-b236-832d1e4c5866> > > Nous vous rappelons que les résultats de Médiamétrie sont et demeurent sa > propriété : ils sont protégés au double > titre du droit d'auteur et de la protection des bases de données. Ce > message est confidentiel et établi à > l'intention de ses destinataires. Tout message électronique étant > susceptible d'altération, la société Médiamétrie > décline toute responsabilité s'il a été altéré, déformé ou falsifié. > > We remind you that the results produced by Médiamétrie are and remain its > sole property covered by both copyright > and databases protection. This message is confidential and intended solely > for the adressees. E-mails are susceptible > to alteration. Neither Médiamétrie company shall be liable for the message > if altered, changed or falsified. > > > > >
