Hello devs, I have found myself in a situation where Spark is doing sub-optimal computations for my RDDs, and I was wondering whether a patch to enable improved performance for this scenario would be a welcome addition to Spark or not.
The scenario happens when trying to cogroup two RDDs that are sorted by key and share the same partitioner. CoGroupedRDD will correctly detect that the RDDs have the same partitioner and will therefore create narrow cogroup split dependencies, as opposed to shuffle dependencies. This is great because it prevents any shuffling from happening. However, the cogroup is unable to detect that the RDDs are sorted in the same way, and will still insert all elements of the RDD in a map in order to join the elements with the same key. When both RDDs are sorted using the same order, the cogroup can just join by doing a single pass over the data (since the data is ordered by key, you can just keep iterating until you find a different key). This would greatly reduce the memory requirements for these kind of operations. Adding this to spark would require adding an “ordering” member to RDD of type Option[Ordering], similarly to how the “partitioner” field works. That way, the sorting operations could populate this field and the operations that could benefit from this knowledge (cogroup, join, groupbykey, etc.) could read it to change their behavior accordingly. Do you think this would be a good addition to Spark? Thanks, Ximo ________________________________ Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción. The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it. Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
