Hey Giovanni, No way to configure the sort order in the current SecondarySort API, so yeah, it's always ascending and you can design around it as you see fit. I think the only reason it's not public is the (lazy) assumption that anything easy to secondary sort on can be re-structured so that sorting in ascending order "works"; the most common secondary sort keys are numeric (often time), which is obviously an easy case; something like a string would require some weird character re-mapping that is probably the thing you're trying to avoid here?
J On Wed, May 17, 2017 at 7:56 AM, Giovanni GATTI PINHEIRO < [email protected]> wrote: > Well, the documentation doesn’t say that it is possible to apply a custom > data sorting, but it doesn’t say either that the sorting will ascendant > only. > > > > Then, we come back to the original question: is it possible to apply a > custom sorting? If not, can I assume as true that sorting is always in > ascendant order and design my entire pipeline based on this assumption? > > > > What am I missing here? > > > > Thanks! > > > > *Giovanni GATTI PINHEIRO* > *Ingénieur Développement* > Volt Europe pour Médiamétrie > > > > www.mediametrie.fr > <https://mediametrie.letsignit.com/r/0/c2abef80-ef94-4c44-b4ef-0feb56a51d40/2fc7a8f2-1619-4f0b-9f14-b960c5c7a5fd> > > > > > > *De :* David Ortiz [mailto:[email protected]] > *Envoyé :* mercredi 17 mai 2017 16:14 > > *À :* [email protected] > *Objet :* Re: Custom SecondarySort > > > > Will the secondarysort api allow you to do anything for this, or will that > only do ascending? > > > > https://crunch.apache.org/user-guide.html#secsort > > https://crunch.apache.org/apidocs/0.10.0/org/apache/ > crunch/lib/SecondarySort.html > > > > On Wed, May 17, 2017 at 10:06 AM Giovanni GATTI PINHEIRO < > [email protected]> wrote: > > This is the first thing that crossed my mind, but it won’t work (at least > using Avro reflection). I think that the reason is that Crunch will not > deserialize the record to perform sorting and he will instead use a > built-in Avro function to do so. As a result, compareTo will have no > effect. This is pure guessing, I don’t have deep Avro-Crunch knowledge. > > > > What I did as work-around was to load all records in-memory and sort them > by myself. It works when you have a small set of records “per grouping > key”, but it won’t work with a large set (or with high memory constraints). > > > > So, I still quite curious if there is a “clean” solution to this or not. > > > > Thank you, > > > > *Giovanni GATTI PINHEIRO* > *Ingénieur Développement* > Volt Europe pour Médiamétrie > > > > www.mediametrie.fr > <https://mediametrie.letsignit.com/r/0/de904011-b6f5-434d-9596-03f0e4cbe9c0/ddffef52-9bfb-412f-80c2-ecc6903d7566> > > > > > > *De :* Dmitry Gorbatsevich [mailto:DmitryGorbatsevich@ > coherentsolutions.com] > *Envoyé :* mercredi 17 mai 2017 15:11 > *À :* [email protected] > *Objet :* Re: Custom SecondarySort > > > > Hey, I have never tried this before, but just curious wether it is > possible to do by overriding compareTo method of your "target fields" > class. > > > > Thanks, > > Dmitry > > > > *From: *Giovanni GATTI PINHEIRO <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Wednesday, May 17, 2017 at 14:52 > *To: *"[email protected]" <[email protected]> > *Subject: *Custom SecondarySort > > > > Hello, > > > > I would like to know if there is an way to implement a custom secondary > sort, that is, instead of applying ascendant sorting on all fields target > fields, I would like to perform something else. If yes, how could that be > done? I couldn’t find how/if it could be done in Crunch documentation. > > > > Thank you in advance. > > > > Best Regards, > > > > *Giovanni GATTI PINHEIRO* > *Ingénieur Développement* > Volt Europe pour Médiamétrie > > > > Email : [email protected] > Buropolis Les Bouillides - 1240, Route des Dolines > 06560 Valbonne Sophia-Antipolis - France > > www.mediametrie.fr > <https://mediametrie.letsignit.com/r/0/5d5cef9f-1c0c-47fc-916f-f206f054bce8/462cef18-f2ef-483c-adc8-0c4d30473036> > > > > > <https://mediametrie.letsignit.com/r/5/801a3736-b2f8-4a60-a04a-a5669668bf74/829465c3-29f3-48c5-b236-832d1e4c5866> > > Nous vous rappelons que les résultats de Médiamétrie sont et demeurent sa > propriété : ils sont protégés au double > titre du droit d'auteur et de la protection des bases de données. Ce > message est confidentiel et établi à > l'intention de ses destinataires. Tout message électronique étant > susceptible d'altération, la société Médiamétrie > décline toute responsabilité s'il a été altéré, déformé ou falsifié. > > We remind you that the results produced by Médiamétrie are and remain its > sole property covered by both copyright > and databases protection. This message is confidential and intended solely > for the adressees. E-mails are susceptible > to alteration. Neither Médiamétrie company shall be liable for the message > if altered, changed or falsified. > > > > > >
