On Thu, Mar 25, 2021 at 12:55 PM Robert Bradshaw
wrote:
> On Wed, Mar 24, 2021 at 7:29 PM Vincent Marquez
> wrote:
>
>>
>> *~Vincent*
>>
>>
>> On Wed, Mar 24, 2021 at 6:07 PM Kenneth Knowles wrote:
>>
>>> The reason I was checking out the code is that sometimes a natural thing
>>> to output wou
On Wed, Mar 24, 2021 at 7:29 PM Vincent Marquez
wrote:
>
> *~Vincent*
>
>
> On Wed, Mar 24, 2021 at 6:07 PM Kenneth Knowles wrote:
>
>> The reason I was checking out the code is that sometimes a natural thing
>> to output would be a summary of what was written. So each chunk of writes
>> and the
I think you are right, since "writer.close()” contains a business logic, it
must be moved to @FinishBundle. The same thing about DeleteFn.
I’ll create a Jira for that.
> On 25 Mar 2021, at 00:49, Kenneth Knowles wrote:
>
> Alex's idea sounds good and like what Vincent maybe implemented. I am j
*~Vincent*
On Wed, Mar 24, 2021 at 6:07 PM Kenneth Knowles wrote:
> The reason I was checking out the code is that sometimes a natural thing
> to output would be a summary of what was written. So each chunk of writes
> and the final chunk written in @FinishBundle. This is, for example, what
> S
The reason I was checking out the code is that sometimes a natural thing to
output would be a summary of what was written. So each chunk of writes and
the final chunk written in @FinishBundle. This is, for example, what SQL
engines do (output # of rows written).
You could output both the summary a
Yeah, the entire input is not always what is needed, and can generally be
achieved via
input -> wait(side input of write) -> do something with the input
Of course one could also do
entire_input_as_output_of_wait -> MapTo(KV.of(null, null)) ->
CombineGlobally(TrivialCombineFn)
to reduce this
I believe that the Wait transform turns this output into a side input, so
outputting the input PCollection might be problematic.
On Wed, Mar 24, 2021 at 4:49 PM Kenneth Knowles wrote:
> Alex's idea sounds good and like what Vincent maybe implemented. I am just
> reading really quickly so sorry i
Alex's idea sounds good and like what Vincent maybe implemented. I am just
reading really quickly so sorry if I missed something...
Checking out the code for the WriteFn I see a big problem:
@Setup
public void setup() {
writer = new Mutator<>(spec, Mapper::saveAsync, "writes");
How about a PCollection containing every element which was successfully
written?
Basically the same things which were passed into it.
Then you could act on every element after its been successfully written to
the sink.
On Wed, Mar 24, 2021 at 3:16 PM Robert Bradshaw wrote:
> On Wed, Mar 24, 202
On Wed, Mar 24, 2021 at 2:36 PM Ismaël Mejía wrote:
> +dev
>
> Since we all agree that we should return something different than
> PDone the real question is what should we return.
>
My proposal is that one returns a PCollection that consists, internally,
of something contentless like nulls. Thi
+dev
Since we all agree that we should return something different than
PDone the real question is what should we return.
As a reminder we had a pretty interesting discussion about this
already in the past but uniformization of our return values has not
happened.
This thread is worth reading for Vi
11 matches
Mail list logo
