Hi Spark Devs,
I am reaching out to gain understanding for accessing data source
CustomMetric in Java.
CustomTaskMetric documentation points out that;
"The final result will be shown up in the data source scan operator in
Spark UI."
SQLAppStatusListenerSuite in the test cases uses physicalPlan t
+1 (for what it's worth). It will definitely help our efforts.
On Fri, Mar 12, 2021 at 12:14 PM Gengliang Wang wrote:
>
> +1 (non-binding)
>
> On Fri, Mar 12, 2021 at 3:00 PM Hyukjin Kwon wrote:
>>
>> +1
>>
>> 2021년 3월 12일 (금) 오후 2:54, Jungtaek Lim 님이 작성:
>>>
>>> +1 (non-binding) Excellent descr
I tried to create a data source, however our use case is bit hard as
we do only know the available offsets within the tasks, not on the
driver. I therefore planned to use accumulators in the
InputPartitionReader but they seem not to work.
Example accumulation is done here
https://github.com/kortem
