Well, the documentation doesn’t say that it is possible to apply a custom data sorting, but it doesn’t say either that the sorting will ascendant only.
Then, we come back to the original question: is it possible to apply a custom sorting? If not, can I assume as true that sorting is always in ascendant order and design my entire pipeline based on this assumption? What am I missing here? Thanks! Giovanni GATTI PINHEIRO Ingénieur Développement Volt Europe pour Médiamétrie www.mediametrie.fr<https://mediametrie.letsignit.com/r/0/c2abef80-ef94-4c44-b4ef-0feb56a51d40/undefined> De : David Ortiz [mailto:[email protected]] Envoyé : mercredi 17 mai 2017 16:14 À : [email protected] Objet : Re: Custom SecondarySort Will the secondarysort api allow you to do anything for this, or will that only do ascending? https://crunch.apache.org/user-guide.html#secsort https://crunch.apache.org/apidocs/0.10.0/org/apache/crunch/lib/SecondarySort.html On Wed, May 17, 2017 at 10:06 AM Giovanni GATTI PINHEIRO <[email protected]<mailto:[email protected]>> wrote: This is the first thing that crossed my mind, but it won’t work (at least using Avro reflection). I think that the reason is that Crunch will not deserialize the record to perform sorting and he will instead use a built-in Avro function to do so. As a result, compareTo will have no effect. This is pure guessing, I don’t have deep Avro-Crunch knowledge. What I did as work-around was to load all records in-memory and sort them by myself. It works when you have a small set of records “per grouping key”, but it won’t work with a large set (or with high memory constraints). So, I still quite curious if there is a “clean” solution to this or not. Thank you, Giovanni GATTI PINHEIRO Ingénieur Développement Volt Europe pour Médiamétrie www.mediametrie.fr<https://mediametrie.letsignit.com/r/0/de904011-b6f5-434d-9596-03f0e4cbe9c0/ddffef52-9bfb-412f-80c2-ecc6903d7566> De : Dmitry Gorbatsevich [mailto:[email protected]<mailto:[email protected]>] Envoyé : mercredi 17 mai 2017 15:11 À : [email protected]<mailto:[email protected]> Objet : Re: Custom SecondarySort Hey, I have never tried this before, but just curious wether it is possible to do by overriding compareTo method of your "target fields" class. Thanks, Dmitry From: Giovanni GATTI PINHEIRO <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday, May 17, 2017 at 14:52 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Custom SecondarySort Hello, I would like to know if there is an way to implement a custom secondary sort, that is, instead of applying ascendant sorting on all fields target fields, I would like to perform something else. If yes, how could that be done? I couldn’t find how/if it could be done in Crunch documentation. Thank you in advance. Best Regards, Giovanni GATTI PINHEIRO Ingénieur Développement Volt Europe pour Médiamétrie Email : [email protected]<mailto:[email protected]> Buropolis Les Bouillides - 1240, Route des Dolines 06560 Valbonne Sophia-Antipolis - France www.mediametrie.fr<https://mediametrie.letsignit.com/r/0/5d5cef9f-1c0c-47fc-916f-f206f054bce8/462cef18-f2ef-483c-adc8-0c4d30473036> [cid:[email protected]]<https://mediametrie.letsignit.com/r/5/801a3736-b2f8-4a60-a04a-a5669668bf74/829465c3-29f3-48c5-b236-832d1e4c5866> Nous vous rappelons que les résultats de Médiamétrie sont et demeurent sa propriété : ils sont protégés au double titre du droit d'auteur et de la protection des bases de données. Ce message est confidentiel et établi à l'intention de ses destinataires. Tout message électronique étant susceptible d'altération, la société Médiamétrie décline toute responsabilité s'il a été altéré, déformé ou falsifié. We remind you that the results produced by Médiamétrie are and remain its sole property covered by both copyright and databases protection. This message is confidential and intended solely for the adressees. E-mails are susceptible to alteration. Neither Médiamétrie company shall be liable for the message if altered, changed or falsified.
