Hi Aljoscha,
Thanks for your feedback.
Yes, we should add DataStream Providers to the table bridge module.
I think your concerns are right, including the relationship between
DataStream and table.
My understanding is that the parallelism specified by the user is only the
initialization
Thanks for monitoring the release progress and kindly reminding us Robert!
Minor: the below link shows the complete list of existing blockers:
https://issues.apache.org/jira/issues/?filter=12349334
Best Regards,
Yu
On Tue, 13 Oct 2020 at 03:03, Robert Metzger wrote:
> Hi all!
>
> According
Thanks for debugging and resolving the issue and driving the discussion Yun!
For the given solutions, I prefer option 1 (supply another Dockerfile using
jemalloc as default memory allocator) because of the below reasons:
1. It's hard to say jemalloc is always better than ptmalloc (glibc malloc),
Yuan Mei created FLINK-19632:
Summary: Introduce a new ResultPartitionType for Approximate Local
Recovery
Key: FLINK-19632
URL: https://issues.apache.org/jira/browse/FLINK-19632
Project: Flink
Hi Timo,
1a) Deprecation: I'm fine with deprecating the old syntax and only use the
new one in all examples, docs, slides etc.
Will update the FLIP later. We can drop the old syntax at some point in the
future, but may need another discussion.
1b) Time attributes: Are you fine with the
Jingsong Lee created FLINK-19631:
Summary: Comments of DecodingFormatFactory is not clear
Key: FLINK-19631
URL: https://issues.apache.org/jira/browse/FLINK-19631
Project: Flink
Issue Type:
Lsw_aka_laplace created FLINK-19630:
---
Summary: Sink data in [ORC] format into Hive By using Legacy Table
API caused unexpected Exception
Key: FLINK-19630
URL: https://issues.apache.org/jira/browse/FLINK-19630
shizhengchao created FLINK-19629:
Summary: English words are spelled incorrectly and an example is
not provided
Key: FLINK-19629
URL: https://issues.apache.org/jira/browse/FLINK-19629
Project: Flink
Caizhi Weng created FLINK-19628:
---
Summary: Introduce multi-input operator for streaming
Key: FLINK-19628
URL: https://issues.apache.org/jira/browse/FLINK-19628
Project: Flink
Issue Type:
Caizhi Weng created FLINK-19627:
---
Summary: Introduce multi-input operator for batch
Key: FLINK-19627
URL: https://issues.apache.org/jira/browse/FLINK-19627
Project: Flink
Issue Type: Sub-task
Caizhi Weng created FLINK-19626:
---
Summary: Introduce multi-input operator construction algorithm
Key: FLINK-19626
URL: https://issues.apache.org/jira/browse/FLINK-19626
Project: Flink
Issue
Caizhi Weng created FLINK-19625:
---
Summary: Introduce multi-input exec node
Key: FLINK-19625
URL: https://issues.apache.org/jira/browse/FLINK-19625
Project: Flink
Issue Type: Sub-task
Caizhi Weng created FLINK-19624:
---
Summary: Update deadlock break-up algorithm to cover more cases
Key: FLINK-19624
URL: https://issues.apache.org/jira/browse/FLINK-19624
Project: Flink
Issue
Caizhi Weng created FLINK-19623:
---
Summary: Introduce ExecEdge to describe information on input edges
for ExecNode
Key: FLINK-19623
URL: https://issues.apache.org/jira/browse/FLINK-19623
Project: Flink
宋洪亮 created FLINK-19622:
---
Summary: flinksql 1.11版本针对avro格式中Map类型value值的空指针异常
Key: FLINK-19622
URL: https://issues.apache.org/jira/browse/FLINK-19622
Project: Flink
Issue Type: Bug
Caizhi Weng created FLINK-19621:
---
Summary: Introduce Multi-input operator in Blink planner
Key: FLINK-19621
URL: https://issues.apache.org/jira/browse/FLINK-19621
Project: Flink
Issue Type:
Tzu-Li (Gordon) Tai created FLINK-19620:
---
Summary: Merge StateFun's ExactlyOnceE2E and RemoteModuleE2E
Key: FLINK-19620
URL: https://issues.apache.org/jira/browse/FLINK-19620
Project: Flink
hailong wang created FLINK-19619:
Summary: Test failed in Azure For EmulatedPubSubSourceTest
Key: FLINK-19619
URL: https://issues.apache.org/jira/browse/FLINK-19619
Project: Flink
Issue
hailong wang created FLINK-19618:
Summary: Boken link in docs
Key: FLINK-19618
URL: https://issues.apache.org/jira/browse/FLINK-19618
Project: Flink
Issue Type: Bug
Components:
Matthias created FLINK-19617:
Summary: Add Metaspace metric
Key: FLINK-19617
URL: https://issues.apache.org/jira/browse/FLINK-19617
Project: Flink
Issue Type: Sub-task
Reporter:
Hi devs,
I would like to start the discussion about supporting multiple input for
blink planner.
As FLIP-92[1] introduces multiple-input stream operator, we can merge the
operators connected by forward shuffle into multiple input operator. So
that the network shuffle can be changed to local
Hi Jark,
1a) Deprecation: I totally agree that dropping old syntax is not very
easy in SQL. That's why I am also extremely cautious about adding new
syntax in this FLIP. However, I would already deprecate the old syntax
and only use the new one in all examples, docs, slides etc. This
Some more thoughts.
I think the standard syntax is what we must provide, but simplifying the
PTF (Polymorphic Table Functions)
syntax is really an interesting topic. This can help to make the new
syntax easier to be picked up by users.
I will share more with what I found. Feel free to join the
@Chesnay Schepler Off the top of my head, I cannot find an easy way
to migrate from the BucketingSink to the StreamingFileSink. It may be
possible but it will require some effort because the logic would be
"read the old state, commit it, and start fresh with the
StreamingFileSink."
On Tue, Oct
On 13.10.20 14:01, David Anderson wrote:
I thought this was waiting on FLIP-46 -- Graceful Shutdown Handling -- and
in fact, the StreamingFileSink is mentioned in that FLIP as a motivating
use case.
Ah yes, I see FLIP-147 as a more general replacement for FLIP-46. Thanks
for the reminder, we
> The BucketingSink suffers from the same problem. It's caused by the fact
> that we don't do a "final" checkpoint before shutting down a pipeline.
> We're trying to resolve that with FLIP-147 [1].
I thought this was waiting on FLIP-46 -- Graceful Shutdown Handling -- and
in fact, the
Hi,
I share a concern:
Although we now support ORC Writer. It's not easy to support. We need to
override something for ORC classes.
Note that we are using a newer version of ORC, which is not forward
compatible. Therefore, the data written by users using Flink Orc writer may
not be readable by
How easy is the migration to the StreamingFileSink?
On 10/13/2020 1:01 PM, Aljoscha Krettek wrote:
On 13.10.20 11:18, David Anderson wrote:
I think the pertinent question is whether there are interesting cases
where
the BucketingSink is still a better choice. One case I'm not sure
about is
On 13.10.20 11:18, David Anderson wrote:
I think the pertinent question is whether there are interesting cases where
the BucketingSink is still a better choice. One case I'm not sure about is
the situation described in docs for the StreamingFileSink under Important
Note 2 [1]:
... upon
Hi everyone,
Timo just raised a good point in the vote thread. I copied the feedback
here:
> Timo:
1) I think we should not offer 2 different kinds of syntax that do the
same thing. We should deprecate the old syntax.
2) We should have session windows in the new syntax as well to give
Piotr Nowojski created FLINK-19616:
--
Summary: Flink : Formats : Parquet compilation failure
Key: FLINK-19616
URL: https://issues.apache.org/jira/browse/FLINK-19616
Project: Flink
Issue
Hi Jark,
thanks for considering my feedback. Yes, let's continue the discussion
in the DISCUSS thread. I will read a bit more about polymorphic table
functions and will reply there again.
Thanks,
Timo
On 13.10.20 11:19, Jark Wu wrote:
Hi Timo,
Sorry, I misremembered the deadline for
Thanks for starting this discussion Yun Gao,
I have three comments/questions:
1) When restarting all tasks independent of the status at checkpoint time
(finished, running, scheduled), we might allocate more resources than we
actually need to run the remaining job. From a scheduling perspective
I think the pertinent question is whether there are interesting cases where
the BucketingSink is still a better choice. One case I'm not sure about is
the situation described in docs for the StreamingFileSink under Important
Note 2 [1]:
... upon normal termination of a job, the last
Hi Timo,
Sorry, I misremembered the deadline for voting and didn't count weekends
in.
Let's open for the voting until 15th Oct (UTC).
Thank you for the valuable feedback. What about continuing
the discussion in the DISUCSS thread?
We can start a new vote if the FLIP needs some modifications.
Dian Fu created FLINK-19615:
---
Summary: HBaseConnectorITCase.testTableSink is instable
Key: FLINK-19615
URL: https://issues.apache.org/jira/browse/FLINK-19615
Project: Flink
Issue Type: Bug
Given that it has been deprecated for three releases now, I am +1 to
dropping it.
On Mon, Oct 12, 2020 at 9:38 PM Chesnay Schepler wrote:
> Is there a way for us to change the module (in a reasonable way) that
> would allow users to continue using it?
> Is it an API problem, or one of
Hi Jingsong,
I'm sorry, I didn't want to block you for so long on this. I thought
about it again.
I think it's fine to add a DataStream Provider if this really unblocks
users from migrating to newer Flink versions. I'm guessing you will add
that to the table bridge module?
Regarding the
Hi Jark,
as far as I can see it is too early to close the voting process. The
official end time is in roughly 1 hour.
Also, usually we exclude weekends from the voting period as you can see
here [1] and here [2] to give people more time to respond discussions.
But this is more of an
Yingjie Cao created FLINK-19614:
---
Summary: Further optimization of sort-merge based blocking shuffle
Key: FLINK-19614
URL: https://issues.apache.org/jira/browse/FLINK-19614
Project: Flink
Jingsong Lee created FLINK-19613:
Summary: Create flink-connector-files-test-utils for formats
testing
Key: FLINK-19613
URL: https://issues.apache.org/jira/browse/FLINK-19613
Project: Flink
Jark Wu created FLINK-19612:
---
Summary: Support CUMULATE window table function in planner
Key: FLINK-19612
URL: https://issues.apache.org/jira/browse/FLINK-19612
Project: Flink
Issue Type: Sub-task
Jark Wu created FLINK-19611:
---
Summary: Introduce WindowProperties MetadataHandler to propagate
window properties
Key: FLINK-19611
URL: https://issues.apache.org/jira/browse/FLINK-19611
Project: Flink
Jark Wu created FLINK-19610:
---
Summary: Support streaming window TopN in planner
Key: FLINK-19610
URL: https://issues.apache.org/jira/browse/FLINK-19610
Project: Flink
Issue Type: Sub-task
Jark Wu created FLINK-19609:
---
Summary: Support streaming window join in planner
Key: FLINK-19609
URL: https://issues.apache.org/jira/browse/FLINK-19609
Project: Flink
Issue Type: Sub-task
Jark Wu created FLINK-19608:
---
Summary: Support window TVF based window aggreagte in planner
Key: FLINK-19608
URL: https://issues.apache.org/jira/browse/FLINK-19608
Project: Flink
Issue Type:
Jark Wu created FLINK-19606:
---
Summary: Implement streaming window join operator
Key: FLINK-19606
URL: https://issues.apache.org/jira/browse/FLINK-19606
Project: Flink
Issue Type: Sub-task
Jark Wu created FLINK-19607:
---
Summary: Implement streaming window TopN operator
Key: FLINK-19607
URL: https://issues.apache.org/jira/browse/FLINK-19607
Project: Flink
Issue Type: Sub-task
Jark Wu created FLINK-19605:
---
Summary: Implement cumulative windowing for window aggregate
operator
Key: FLINK-19605
URL: https://issues.apache.org/jira/browse/FLINK-19605
Project: Flink
Issue
Jark Wu created FLINK-19604:
---
Summary: FLIP-145: Support SQL windowing table-valued function
Key: FLINK-19604
URL: https://issues.apache.org/jira/browse/FLINK-19604
Project: Flink
Issue Type: New
Yingjie Cao created FLINK-19603:
---
Summary: Introduce shuffle data compression to sort-merge based
blocking shuffle
Key: FLINK-19603
URL: https://issues.apache.org/jira/browse/FLINK-19603
Project: Flink
Hi all
Users report they meet serious memory leak when submitting jobs continously in
session mode within k8s (please refer to FLINK-18712[1] ), and I also reproduce
this to find this is caused by memory fragmentation of glibc [2][3] and provide
solutions to fix this:
* Quick but not very
Hi Arvid,
Very thanks for the comments!
>>> 4) Yes, the interaction is not trivial and also I have not completely
>>> thought it through. But in general, I'm currently at the point where I
>>> think that we also need non-checkpoint related events in unaligned
>>> checkpoints. So just keep that
+1
Best,
Shawn Huang
hailongwang <18868816...@163.com> 于2020年10月12日周一 下午11:21写道：
> +1
> Best,
> Hailong Wang
> At 2020-10-12 17:00:34, "Xintong Song" wrote:
> >+1
> >
> >Thank you~
> >
> >Xintong Song
> >
> >
> >
> >On Mon, Oct 12, 2020 at 5:59 PM Jark Wu wrote:
> >
> >> +1
> >>
> >> On Mon,
Yingjie Cao created FLINK-19602:
---
Summary: Introduce new config options to enable sort-merge based
blocking shuffle
Key: FLINK-19602
URL: https://issues.apache.org/jira/browse/FLINK-19602
Project:
Hi all,
The voting time for FLIP-145 has passed. I'm closing the vote now.
There were 6 +1 votes, 3 of which are binding:
- Benchao (binding)
- Jark (binding)
- Pengcheng Liu
- Dalong Liu (刘大龙)
- Hailong Wang
- Jingsong (binding)
There were no disapproving votes.
Thus, FLIP-145 has been
Yingjie Cao created FLINK-19601:
---
Summary: Introduce sort-merge based blocking result partition
SortMergeResultPartition and the corresponding subpartition reader
Key: FLINK-19601
URL:
Yingjie Cao created FLINK-19600:
---
Summary: Introduce PartitionedFile and the corresponding
writer/reader for sort-merge based blocking shuffle
Key: FLINK-19600
URL: https://issues.apache.org/jira/browse/FLINK-19600
Jingsong Lee created FLINK-19599:
Summary: Introduce ParquetBulkFormatFactory to integrate new
FileSource to table
Key: FLINK-19599
URL: https://issues.apache.org/jira/browse/FLINK-19599
Project:
Zhu Zhu created FLINK-19598:
---
Summary: E2E test Kerberized YARN application on Docker test
(default input) fails on CI
Key: FLINK-19598
URL: https://issues.apache.org/jira/browse/FLINK-19598
Project: Flink
60 matches
Mail list logo
