Re: Need help on orcsourcetable with hdfs

2019-10-15 Thread Pritam Sadhukhan
Can anyone please help me with the conf files? Am I missing anything on the configuration part? Regards, Pritam. On Tue, 15 Oct 2019 at 08:48, Pritam Sadhukhan wrote: > Thanks for the information. > > I am able to see all the files using hdfs shell command. > Even I am able to pull the data on

[ARM support] Travis ARM CI is now in Alpha Release

2019-10-15 Thread Xiyuan Wang
Hi all, Recently Travis announced that ARM arch is in Alpha release[1]. Since Flink has integrated with Travis already, I think it's quite easy for Flink to use it for ARM CI. Maybe some of you know that I'm working on Flink ARM testing and support. I suggested to use OpenLab[2] as the ARM CI

[jira] [Created] (FLINK-14402) KafkaProducerAtLeastOnceITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator:214->KafkaProducerTestBase.testOneToOneAtLeastOnce failed on Travis

2019-10-15 Thread vinoyang (Jira)
vinoyang created FLINK-14402: Summary: KafkaProducerAtLeastOnceITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator:214->KafkaProducerTestBase.testOneToOneAtLeastOnce failed on Travis Key: FLINK-14402

Re: [Discussion] FLIP-79 Flink Function DDL Support

2019-10-15 Thread Xuefu Z
Thanks to Peter for the proposal! I left some comments in the google doc. Besides what Bowen pointed out, I'm unclear about how things work end to end from the document. For instance, SQL DDL-like function definition is mentioned. I guess just having a DDL for it doesn't explain how it's

[jira] [Created] (FLINK-14401) create DefaultCatalogFunctionFactory to instantiate regular java class-based udf

2019-10-15 Thread Bowen Li (Jira)
Bowen Li created FLINK-14401: Summary: create DefaultCatalogFunctionFactory to instantiate regular java class-based udf Key: FLINK-14401 URL: https://issues.apache.org/jira/browse/FLINK-14401 Project:

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

2019-10-15 Thread Flavio Pompermaier
Definitely on the same page..+1 to keep it in a separate repo (at least until the cose becomes "stable" and widely adopted from the community) Il Mar 15 Ott 2019, 23:17 Stephan Ewen ha scritto: > Hi Flink folks! > > After the positive reaction to the contribution proposal for Stateful >

[DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

2019-10-15 Thread Stephan Ewen
Hi Flink folks! After the positive reaction to the contribution proposal for Stateful Functions, I would like to kick off the discussion for the big question: In which form should it go into Flink? Before jumping into the "repository" question directly, let's get some clarity on what would be

Re: [VOTE] Drop Python 2 support for 1.10

2019-10-15 Thread Bowen Li
+1 On Sun, Oct 13, 2019 at 10:54 PM Hequn Cheng wrote: > +1 > > Thanks a lot for driving this, Dian! > > On Mon, Oct 14, 2019 at 1:46 PM jincheng sun > wrote: > > > +1 > > > > Dian Fu 于2019年10月14日周一 下午1:21写道: > > > > > Hi all, > > > > > > I would like to start the vote for "Drop Python 2

[VOTE] FLIP-68: Extend Core Table System with Pluggable Modules

2019-10-15 Thread Bowen Li
Hi all, I'd like to kick off a voting thread for FLIP-68: Extend Core Table System with Pluggable Modules [1], as we have reached consensus in [2]. The voting period will be open for at least 72 hours, ending at 7pm Oct 18 UTC. Thanks, Bowen [1]

Re: [VOTE] FLIP-68: Extend Core Table System with Modular Plugins

2019-10-15 Thread Bowen Li
sorry, please ignore this thread as the FLIP's name should be "Extend Core Table System with Pluggable Modules" On Tue, Oct 15, 2019 at 9:59 AM Bowen Li wrote: > Hi all, > > I'd like to kick off a voting thread for FLIP-68: Extend Core Table System > with Modular Plugins [1], as we have reached

Re: [Discussion] FLIP-79 Flink Function DDL Support

2019-10-15 Thread Bowen Li
Hi Zhenqiu, Thanks for taking on this effort! A couple questions: - Though this FLIP is about function DDL, can we also think about how the created functions can be mapped to CatalogFunction and see if we need to modify CatalogFunction interface? Syntax changes need to be backed by the backend.

Re: [VOTE] FLIP-64: Support for Temporary Objects in Table module

2019-10-15 Thread Bowen Li
+1 On Tue, Oct 15, 2019 at 5:09 AM Jark Wu wrote: > +1 from my side. > > Cheers, > Jark > > On Tue, 15 Oct 2019 at 19:11, vino yang wrote: > > > +1 > > > > Best, > > Vino > > > > Aljoscha Krettek 于2019年10月15日周二 下午4:31写道: > > > > > +1 > > > > > > Best, > > > Aljoscha > > > > > > > On 14. Oct

[VOTE] FLIP-68: Extend Core Table System with Modular Plugins

2019-10-15 Thread Bowen Li
Hi all, I'd like to kick off a voting thread for FLIP-68: Extend Core Table System with Modular Plugins [1], as we have reached consensus in [2]. The voting period will be open for at least 72 hours, ending at 5pm Oct 18, UTC. Thanks, Bowen [1]

Re: [DISCUSS] FLIP-68: Extend Core Table System with Modular Plugins

2019-10-15 Thread Bowen Li
Hi all, I have updated the doc to reflect the discussion results. I will start a voting thread shortly. Thanks! Bowen On Fri, Oct 11, 2019 at 12:44 AM Timo Walther wrote: > I'm fine with `type` for consistency reasons for now. I hope we will fix > that when we rework the properties design. >

Re: [VOTE] FLIP-57: Rework FunctionCatalog, latest updated

2019-10-15 Thread Bowen Li
Hi all, I hereby announce the FLIP has passed with 6 +1 votes, 4 binding (Dawid, Timo, Aljoscha, Jark) and 2 non-binding (Xuefu, Jingsong). Thanks for your review and participation! On Thu, Oct 10, 2019 at 1:08 AM Jingsong Li wrote: > +1 > > Best, > Jingsong Lee > > On Thu, Oct 10, 2019 at

Re: [VOTE] FLIP-77: Introduce ConfigOptions with Data Types

2019-10-15 Thread Till Rohrmann
Sorry for the confusion. I should have checked with an external mail client. Thanks a lot for the clarification. Cheers, Till On Tue, Oct 15, 2019 at 2:07 PM Jark Wu wrote: > +1 > > It is a separate [VOTE] thread in my Mail client. > > Best, > Jark > > > 在 2019年10月15日,18:58,Dawid Wysakowicz

[jira] [Created] (FLINK-14400) Shrink the scope of MemoryManager from TaskExecutor to slot

2019-10-15 Thread Andrey Zagrebin (Jira)
Andrey Zagrebin created FLINK-14400: --- Summary: Shrink the scope of MemoryManager from TaskExecutor to slot Key: FLINK-14400 URL: https://issues.apache.org/jira/browse/FLINK-14400 Project: Flink

[jira] [Created] (FLINK-14399) Add memory chunk reservation API to MemoryManager

2019-10-15 Thread Andrey Zagrebin (Jira)
Andrey Zagrebin created FLINK-14399: --- Summary: Add memory chunk reservation API to MemoryManager Key: FLINK-14399 URL: https://issues.apache.org/jira/browse/FLINK-14399 Project: Flink

[jira] [Created] (FLINK-14398) Codegen produced larger-than-64kb method

2019-10-15 Thread Hao Dang (Jira)
Hao Dang created FLINK-14398: Summary: Codegen produced larger-than-64kb method Key: FLINK-14398 URL: https://issues.apache.org/jira/browse/FLINK-14398 Project: Flink Issue Type: Bug

performances of S3 writing with many buckets in parallel

2019-10-15 Thread Enrico Agnoli
Starting here the discussion after an initial discussion with Ververica and AWS teams during FlinkForward. I'm investigating the performances of a Flink job that transports data from Kafka to an S3 Sink. We are using a BucketingSink to write parquet files. The bucketing logic divides the

Re: [DISCUSS] FLIP policy for introducing config option keys

2019-10-15 Thread Zili Chen
The naming concern above can be a separated issue since it looks also affect FLIP-54 and isn't limited for config option changes FLIP. Best, tison. Aljoscha Krettek 于2019年10月15日周二 下午8:37写道: > Another PR that introduces new config options: > https://github.com/apache/flink/pull/9759 > > > On

Re: [DISCUSS] FLIP policy for introducing config option keys

2019-10-15 Thread Jark Wu
Hi all, This is an interesting problem and sometimes it bothers me too. In general, I agree that it should deserve a voting process. But I also share the same concern with Zili that the FLIP number will be flooded. So, I think the compromise way is such a small new interface does not need a

Re: [DISCUSS] FLIP policy for introducing config option keys

2019-10-15 Thread Aljoscha Krettek
Another PR that introduces new config options: https://github.com/apache/flink/pull/9759 > On 15. Oct 2019, at 14:31, Zili Chen wrote: > > Hi Aljoscha & Dawid & Kostas, > > I agree that changes on config option keys deserve a FLIP and it is > reasonable > we commit the changes with a standard

[jira] [Created] (FLINK-14397) Failed to run Hive UDTF with ClassCastException

2019-10-15 Thread Rui Li (Jira)
Rui Li created FLINK-14397: -- Summary: Failed to run Hive UDTF with ClassCastException Key: FLINK-14397 URL: https://issues.apache.org/jira/browse/FLINK-14397 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP policy for introducing config option keys

2019-10-15 Thread Zili Chen
Hi Aljoscha & Dawid & Kostas, I agree that changes on config option keys deserve a FLIP and it is reasonable we commit the changes with a standard FLIP process so that ensure the change given proper visibility. My concern is about naming. Given FLIP-73 as an example, if FLIPs associated to

Re: [DISCUSS] FLIP policy for introducing config option keys

2019-10-15 Thread Kostas Kloudas
Hi Aljoscha, Given that config option keys are user-facing and any change there is breaking, I think there should be a discussion about them and a FLIP, where people have to actually vote for it seems to be the right place. I understand that this is tedious (and actually I will also have to open

Re: [DISCUSS] FLIP policy for introducing config option keys

2019-10-15 Thread Dawid Wysakowicz
Hi Aljoscha, I understand this might be troublesome at times, but I see benefit in having at least 3 +1s from committers on those. In the end config options are part of user facing API. In the end adapting config options is a commitment to support those options in the future. I think having a

Re: [VOTE] FLIP-64: Support for Temporary Objects in Table module

2019-10-15 Thread Jark Wu
+1 from my side. Cheers, Jark On Tue, 15 Oct 2019 at 19:11, vino yang wrote: > +1 > > Best, > Vino > > Aljoscha Krettek 于2019年10月15日周二 下午4:31写道: > > > +1 > > > > Best, > > Aljoscha > > > > > On 14. Oct 2019, at 14:55, Kurt Young wrote: > > > > > > +1 > > > > > > Best, > > > Kurt > > > > > >

Re: [VOTE] FLIP-77: Introduce ConfigOptions with Data Types

2019-10-15 Thread Jark Wu
+1 It is a separate [VOTE] thread in my Mail client. Best, Jark > 在 2019年10月15日,18:58,Dawid Wysakowicz 写道: > > Hi Till, > > It should and it actually is. > > I think it's some feature of gmail that it groups conversations with a > similar topic into a conversation. (I also see it this way

[DISCUSS] FLIP policy for introducing config option keys

2019-10-15 Thread Aljoscha Krettek
Hi Everyone, The title says it all, do you think we need to cover all config options that we introduce/change by FLIPs? I was thinking about this because of the FLIP-73 work, which will introduce some new config options and also because I just spotted a PR [1] that introduces some config

[jira] [Created] (FLINK-14396) Implement rudimentary non-blocking network output

2019-10-15 Thread zhijiang (Jira)
zhijiang created FLINK-14396: Summary: Implement rudimentary non-blocking network output Key: FLINK-14396 URL: https://issues.apache.org/jira/browse/FLINK-14396 Project: Flink Issue Type: Task

Re: [VOTE] FLIP-64: Support for Temporary Objects in Table module

2019-10-15 Thread vino yang
+1 Best, Vino Aljoscha Krettek 于2019年10月15日周二 下午4:31写道: > +1 > > Best, > Aljoscha > > > On 14. Oct 2019, at 14:55, Kurt Young wrote: > > > > +1 > > > > Best, > > Kurt > > > > > > On Fri, Oct 11, 2019 at 1:39 PM Dawid Wysakowicz > > > wrote: > > > >> Hi everyone, > >> I would like to start a

Re: [VOTE] FLIP-74: Flink JobClient API

2019-10-15 Thread vino yang
+1 (non-binding) Best, Vino Aljoscha Krettek 于2019年10月15日周二 下午2:59写道: > +1 (binding) > > Best, > Aljoscha > > > On 15. Oct 2019, at 04:01, Zili Chen wrote: > > > > Hi all, > > > > +1 from my side. > > > > Given the current state of this voting thread, FLIP-74 is accepted > > with 3 binding

Re: [VOTE] FLIP-77: Introduce ConfigOptions with Data Types

2019-10-15 Thread Dawid Wysakowicz
Hi Till, It should and it actually is. I think it's some feature of gmail that it groups conversations with a similar topic into a conversation. (I also see it this way in gmail, but I see it as separate threads in my mail client) You can see that it is a separate thread e.g. in the ML

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

2019-10-15 Thread Chesnay Schepler
I have updated the FLIP. - adopted job-/cluster partitions naming scheme - out-lined interface for new component living in the RM (currently called ThinShuffleMaster, but I'm not a fan of the name. Suggestions would be appreciated) - added a note that the ShuffleService changes are only

Re: [VOTE] FLIP-77: Introduce ConfigOptions with Data Types

2019-10-15 Thread Till Rohrmann
Shouldn't the voting happen in a distinct [VOTE] thread? Cheers, Till On Tue, Oct 15, 2019 at 10:46 AM Kurt Young wrote: > +1 > > Best, > Kurt > > > On Tue, Oct 15, 2019 at 9:30 AM Dawid Wysakowicz > wrote: > > > Hi everyone, > > I would like to start a vote on FLIP-77. > > > > Please vote

Re: [VOTE] FLIP-78: Flink Python UDF Environment and Dependency Management

2019-10-15 Thread Wei Zhong
Hi Thomas, Thanks a lot for your suggestion! As you can see from the section "Goals" that this FLIP focuses on the dependency management in process mode. However, the APIs and design proposed in this FLIP also applies for the docker mode. So it makes sense to me to also describe how this

Re: [VOTE] FLIP-77: Introduce ConfigOptions with Data Types

2019-10-15 Thread Kurt Young
+1 Best, Kurt On Tue, Oct 15, 2019 at 9:30 AM Dawid Wysakowicz wrote: > Hi everyone, > I would like to start a vote on FLIP-77. > > Please vote for the following design document: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-77%3A+Introduce+ConfigOptions+with+Data+Types > > >

Re: [VOTE] FLIP-64: Support for Temporary Objects in Table module

2019-10-15 Thread Aljoscha Krettek
+1 Best, Aljoscha > On 14. Oct 2019, at 14:55, Kurt Young wrote: > > +1 > > Best, > Kurt > > > On Fri, Oct 11, 2019 at 1:39 PM Dawid Wysakowicz > wrote: > >> Hi everyone, >> I would like to start a vote on FLIP-64. The discussion seems to have >> reached an agreement. >> >> Please vote

Re: [DISCUSS] Improve Flink logging with contextual information

2019-10-15 Thread Paul Lam
+1 to Rong’s approach. We use a similar solution to the log context problem on YARN setups. FYI. WRT container contextual informations, we collection logs via ELK so that the log file paths (which contains application id and container id) and the host are attached with the logs. But if you

[jira] [Created] (FLINK-14395) Refactor ES 6 connector to split table-specific code into flink-sql-connector-elasticsearch6

2019-10-15 Thread vinoyang (Jira)
vinoyang created FLINK-14395: Summary: Refactor ES 6 connector to split table-specific code into flink-sql-connector-elasticsearch6 Key: FLINK-14395 URL: https://issues.apache.org/jira/browse/FLINK-14395

[VOTE] FLIP-77: Introduce ConfigOptions with Data Types

2019-10-15 Thread Dawid Wysakowicz
Hi everyone, I would like to start a vote on FLIP-77. Please vote for the following design document: https://cwiki.apache.org/confluence/display/FLINK/FLIP-77%3A+Introduce+ConfigOptions+with+Data+Types The discussions can be found at:

Re: [DISCUSS] FLIP-77: Introduce ConfigOptions with Data Types

2019-10-15 Thread Dawid Wysakowicz
Good idea Aljosha, I will kick off a voting thread shortly as there were no comments in this thread so far, so I would assume there are no objections ;) Best, Dawid On 15/10/2019 09:21, Aljoscha Krettek wrote: > This looks very slim now! If there are no objections on this I would propose > we

Re: [DISCUSS] FLIP-77: Introduce ConfigOptions with Data Types

2019-10-15 Thread Aljoscha Krettek
This looks very slim now! If there are no objections on this I would propose we quickly move on to a VOTE thread. Best, Aljoscha > On 11. Oct 2019, at 10:35, Timo Walther wrote: > > Hi everyone, > > as mentioned in the [DISCUSS] thread of FLIP-54 [1]. The evolution of > ConfigOption is a

Re: [VOTE] FLIP-74: Flink JobClient API

2019-10-15 Thread Aljoscha Krettek
+1 (binding) Best, Aljoscha > On 15. Oct 2019, at 04:01, Zili Chen wrote: > > Hi all, > > +1 from my side. > > Given the current state of this voting thread, FLIP-74 is accepted > with 3 binding vote and 2 non-binding vote. Thanks for your > participation! > > I will update the wiki to

[jira] [Created] (FLINK-14394) Remove unnecessary notifySubpartitionConsumed method from view reader

2019-10-15 Thread zhijiang (Jira)
zhijiang created FLINK-14394: Summary: Remove unnecessary notifySubpartitionConsumed method from view reader Key: FLINK-14394 URL: https://issues.apache.org/jira/browse/FLINK-14394 Project: Flink