joosthooz commented on PR #33738: URL: https://github.com/apache/arrow/pull/33738#issuecomment-1400316462
Hi, I gave this branch a spin, and it seems that the nesting has become inconsistent:  There's 2 ReadBatch spans under InitialTask. 1 of these has all the FragmentsToBatches as its child spans (these were nested under the SourceNode before). The other keeps recursively nesting more ReadBatch spans. Each has a ProcessMorsel, that has the filter, project and sink spans nested under each other. Then the dataset writer also keeps nesting WriteAndCheckBackpressure.  Is there a way to go back to making most of these spans siblings again? Do we want to change the organization of the spans in this PR from having 1 span for each node in the graph, each having a span for every chunk of data it processes (how it was before), to having a ProcessMorsel for each chunk of data, each having a span for each node it traverses through? I think I can help in a follow-up PR, especially for the dataset writer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
