date:20180227

Re: [VOTE] Release 1.4.2, release candidate #2

2018-02-27 Thread Aljoscha Krettek

I think it would be very good to also include the fix for 
https://issues.apache.org/jira/browse/FLINK-8798 
. Some users have already run 
into this problem and popular Web Containers that use child-first classloading 
also all have an exception for commons-logging. What do you think? And sorry 
for the inconvenience.

> On 27. Feb 2018, at 13:30, Tzu-Li (Gordon) Tai  wrote:
> 
> Hi everyone,
> 
> Please review and vote on release candidate #1 for Flink 1.4.2, as follows: 
> [ ] +1, Approve the release 
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> The complete staging area is available for your review, which includes: 
> * JIRA release notes [1], 
> * the official Apache source release and binary convenience releases to be 
> deployed to dist.apache.org [2], which are signed with the key with 
> fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3], 
> * all artifacts to be deployed to the Maven Central Repository [4], 
> * source code tag “release-1.4.2-rc1” [5], 
> * website pull request listing the new release [6]. 
> * A complete list of all new commits in release-1.4.2-rc1, since 
> release-1.4.1 [7]
> 
> The vote will be open for at least 72 hours.
> Please test the release and vote for the release candidate before Friday 
> (March 2nd), 7pm CET.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
> 
> Thanks, 
> Gordon
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12342745
> [2] http://people.apache.org/~tzulitai/flink-1.4.2-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1148
> [5] 
> https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=d5ef96aeb63ad3dd4d0676e0c32a35e888ecb409
> [6] https://github.com/apache/flink-web/pull/102
> [7]
> * b74c705157 - [FLINK-8543] Don’t call super.close() in 
> AvroKeyValueSinkWriter (16 hours ago)
> * a0193f10af - [FLINK-8772] [kafka] Fix missing log parameter (21 hours ago)
> * 0396fc8c03 - [hotfix] [test] Also trap INT signal in Kafka end-to-end test 
> (21 hours ago)
> * 59169d0eeb - [hotfix] [test] Make test-streaming-kafka010.sh more flexible 
> for local execution (21 hours ago)
> * cb6a8a68c4 - [FLINK-8741] [kafka] Fix incorrect user code classloader in 
> FlinkKafkaConsumer (21 hours ago)
> * 0d89a1c9f1 - [hotfix] Update docs version to Flink 1.4.1 (4 days ago)
> * f06ec38f22 - [FLINK-8735] Add new StatefulJobSavepointMigrationITCase (5 
> days ago)
> * 82e6f8d5b8 - [FLINK-8735] Rename StatefulJobSavepointMigrationITCase (6 
> days ago)
> * a7df42485a - [FLINK-8574][travis] Add timestamp to logging messages (7 days 
> ago)
> * 527faf63e2 - [hotfix][prometheus][tests] Add utility for generating port 
> ranges (7 days ago)
> * d88f43bb06 - [hotfix][prometheus] Document internal usage of 
> CollectorRegistry.defaultRegistry (7 days ago)
> * 45efe4702a - [FLINK-8621][prometheus][tests] Remove 
> endpointIsUnavailableAfterReporterIsClosed() (7 days ago)
> * 528317c8f2 - [FLINK-8692][docs] Remove extra parenthesis in scala code 
> samples (9 days ago)
> * cc76c323f0 - [FLINK-8303] [docs] Allow to overwrite ruby/gem binary (11 
> days ago)
> * d4435e1596 - [FLINK-8303] Add hawkins back to Gemfile (11 days ago)
> * 50b648473f - [FLINK-8308] Remove explicit yajl-ruby dependency, update 
> Jekyll to 3+ (11 days ago)
> * 054af99746 - [FLINK-8520][cassandra] Fix race condition (11 days ago)
> * 1b70f50d93 - [FLINK-8576][QS] Reduce verbosity when classes can’t be found 
> (11 days ago)
> * f2b5635bc8 - [hotfix] Remove more checkState() calls from SharedBuffer 
> serialization (12 days ago)
> * 59f9ded8fd - [FLINK-8652] [QS] Reduce log level in getKvState to DEBUG. (13 
> days ago)
> * a044d9d6ca - [hotfix] Remove costly logging statements from CEP 
> SharedBuffer (13 days ago)
> * bafb91eeb5 - [FLINK-8423] OperatorChain#pushToOperator catch block may fail 
> with NPE (2 weeks ago)

Re: Verifying watermarks in integration test

2018-02-27 Thread Thomas Weise

Hi Xingcan,

thanks, this is a good way of testing an individual operator. I had written
my own mock code to intercept source context and collect the results, this
is a much better approach for operator testing.

I wonder how I can verify with an embedded Flink cluster though. Even
though my single operator test passes, the results are not emitted as
expected within a topology (not observed in the attached sink). What's the
test approach there?

Thanks,
Thomas

On Wed, Feb 21, 2018 at 12:43 AM, Xingcan Cui  wrote:

> Hi Thomas,
>
> some test cases in JoinHarnessTest  flink/blob/release-1.4/flink-libraries/flink-table/src/
> test/scala/org/apache/flink/table/runtime/harness/JoinHarnessTest.scala>
> show how to verify the emitted watermarks.
>
> Hope this helps.
>
> Best,
> Xingcan
>
> > On 21 Feb 2018, at 2:09 PM, Thomas Weise  wrote:
> >
> > Hi,
> >
> > I have a streaming integration test with two operators. A source that
> emits
> > records and watermarks, and a sink that collects the records. The
> topology
> > runs in embedded mode and the results are collected in memory. Now, in
> > addition to the records, I also want to verify that watermarks have been
> > emitted. What's the recommended way of doing that?
> >
> > Thanks
>
>

Re: [DISCUS] Flink SQL Client dependency management

2018-02-27 Thread Stephan Ewen

I think one problem with the "one fat jar for all" is that some
dependencies clash in the classnames across versions:
  - Kafka 0.9, 0.10, 0.11, 1.0
  - Elasticsearch 2, 4, and 5

There are probably others as well...

On Tue, Feb 27, 2018 at 2:57 PM, Timo Walther  wrote:

> Hi Xingcan,
>
> thank you for your feedback. Regarding (3) we also thought about that but
> this approach would not scale very well. Given that we might have fat jars
> for multiple versions (Kafka 0.8, Kafka 0.6 etc.) such an all-in-one
> solution JAR file might easily go beyond 1 or 2 GB. I don't know if users
> want to download that just for a combination of connector and format.
>
> Timo
>
>
> Am 2/27/18 um 2:16 PM schrieb Xingcan Cui:
>
> Hi Timo,
>>
>> thanks for your efforts. Personally, I think the second option would be
>> better and here are my feelings.
>>
>> (1) The SQL client is designed to offer a convenient way for users to
>> manipulate data with Flink. Obviously, the second option would be more
>> easy-to-use.
>>
>> (2) The script will help to manage the dependencies automatically, but
>> with less flexibility. Once the script cannot meet the need, users have to
>> modify it themselves.
>>
>> (3) I wonder whether we could package all these built-in connectors and
>> formats into a single JAR. With this all-in-one solution, users don’t need
>> to consider much about the dependencies.
>>
>> Best,
>> Xingcan
>>
>> On 27 Feb 2018, at 6:38 PM, Stephan Ewen  wrote:
>>>
>>> My first intuition would be to go for approach #2 for the following
>>> reasons
>>>
>>> - I expect that in the long run, the scripts will not be that simple to
>>> maintain. We saw that with all shell scripts thus far: they start simple,
>>> and then grow with many special cases for this and that setup.
>>>
>>> - Not all users have Maven, automatically downloading and configuring
>>> Maven could be an option, but that makes the scripts yet more tricky.
>>>
>>> - Download-and-drop-in is probably still easier to understand for users
>>> than the syntax of a script with its parameters
>>>
>>> - I think it may actually be even simpler to maintain for us, because all
>>> it does is add a profile or build target to each connector to also create
>>> the fat jar.
>>>
>>> - Storage space is no longer really a problem. Worst case we host the fat
>>> jars in an S3 bucket.
>>>
>>>
>>> On Mon, Feb 26, 2018 at 7:33 PM, Timo Walther 
>>> wrote:
>>>
>>> Hi everyone,

 as you may know a first minimum version of FLIP-24 [1] for the upcoming
 Flink SQL Client has been merged to the master. We also merged
 possibilities to discover and configure table sources without a single
 line
 of code using string-based properties [2] and Java service provider
 discovery.

 We are now facing the issue of how to manage dependencies in this new
 environment. It is different from how regular Flink projects are created
 (by setting up a a new Maven project and build a jar or fat jar).
 Ideally,
 a user should be able to select from a set of prepared connectors,
 catalogs, and formats. E.g., if a Kafka connector and Avro format is
 needed, all that should be required is to move a "flink-kafka.jar" and
 "flink-avro.jar" into the "sql_lib" directory that is shipped to a Flink
 cluster together with the SQL query.

 The question is how do we want to offer those JAR files in the future?
 We
 see two options:

 1) We prepare Maven build profiles for all offered modules and provide a
 shell script for building fat jars. A script call could look like
 "./sql-client-dependency.sh kafka 0.10". It would automatically download
 what is needed and place the JAR file in the library folder. This
 approach
 would keep our development effort low but would require Maven to be
 present
 and builds to pass on different environments (e.g. Windows).

 2) We build fat jars for these modules with every Flink release that can
 be hostet somewhere (e.g. Apache infrastructure, but not Maven central).
 This would make it very easy to add a dependency by downloading the
 prepared JAR files. However, it would require to build and host large
 fat
 jars for every connector (and version) with every Flink major and minor
 release. The size of such a repository might grow quickly.

 What do you think? Do you see other options to make adding dependencies
 as
 possible?

 Regards,

 Timo

 [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+
 SQL+Client

 [2] https://issues.apache.org/jira/browse/FLINK-8240

>

[jira] [Created] (FLINK-8801) S3's eventual consistent read-after-write may fail yarn deployment of resources to S3

2018-02-27 Thread Nico Kruber (JIRA)

Nico Kruber created FLINK-8801:
--

 Summary: S3's eventual consistent read-after-write may fail yarn 
deployment of resources to S3
 Key: FLINK-8801
 URL: https://issues.apache.org/jira/browse/FLINK-8801
 Project: Flink
  Issue Type: Bug
  Components: FileSystem, ResourceManager, YARN
Affects Versions: 1.4.0, 1.5.0
Reporter: Nico Kruber
Assignee: Nico Kruber
 Fix For: 1.5.0


According to 
https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel:

{quote}
Amazon S3 provides read-after-write consistency for PUTS of new objects in your 
S3 bucket in all regions with one caveat. The caveat is that if you make a HEAD 
or GET request to the key name (to find if the object exists) before creating 
the object, Amazon S3 provides eventual consistency for read-after-write.
{quote}

Some S3 file system implementations may actually execute such a request for the 
about-to-write object and thus the read-after-write is only eventually 
consistent. {{org.apache.flink.yarn.Utils#setupLocalResource()}} currently 
relies on a consistent read-after-write since it accesses the remote resource 
to get file size and modification timestamp. Since there we have access to the 
local resource, we can use the data from there instead and circumvent the 
problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8800) Set Logging to TRACE for org.apache.flink.runtime.rest.handler.job.metrics.JobVertexMetricsHandler

2018-02-27 Thread Stephan Ewen (JIRA)

Stephan Ewen created FLINK-8800:
---

 Summary: Set Logging to TRACE for 
org.apache.flink.runtime.rest.handler.job.metrics.JobVertexMetricsHandler
 Key: FLINK-8800
 URL: https://issues.apache.org/jira/browse/FLINK-8800
 Project: Flink
  Issue Type: Bug
  Components: REST
Reporter: Stephan Ewen
 Fix For: 1.5.0, 1.6.0


When setting the log level to {{DEBUG}}, the logs are swamped with statements 
as below, making it hard to read the debug logs.

{code}
2018-02-22 13:41:04,016 DEBUG 
org.apache.flink.runtime.rest.handler.job.metrics.JobVertexMetricsHandler  - 
Received request 
/jobs/ec1c9d7a3c413a9523656efa58735009/vertices/ded95c643b42f31cf882a8986207fd30/metrics?get=0.currentLowWatermark.
2018-02-22 13:41:04,048 DEBUG 
org.apache.flink.runtime.rest.handler.job.metrics.JobVertexMetricsHandler  - 
Received request 
/jobs/ec1c9d7a3c413a9523656efa58735009/vertices/eec5890dac9c38f66954443809beb5b0/metrics?get=0.currentLowWatermark.
2018-02-22 13:41:04,052 DEBUG 
org.apache.flink.runtime.rest.handler.job.metrics.JobVertexMetricsHandler  - 
Received request 
/jobs/ec1c9d7a3c413a9523656efa58735009/vertices/2a964ee72788c82cb7d15e352d9a94f6/metrics?get=0.currentLowWatermark.
2018-02-22 13:41:04,079 DEBUG 
org.apache.flink.runtime.rest.handler.job.metrics.JobVertexMetricsHandler  - 
Received request 
/jobs/ec1c9d7a3c413a9523656efa58735009/vertices/1d9c83f6e1879fdbe461aafac16eb8a5/metrics?get=0.currentLowWatermark.
2018-02-22 13:41:04,085 DEBUG 
org.apache.flink.runtime.rest.handler.job.metrics.JobVertexMetricsHandler  - 
Received request 
/jobs/ec1c9d7a3c413a9523656efa58735009/vertices/4063620891a151092c5bcedb218870a6/metrics?get=0.currentLowWatermark.
2018-02-22 13:41:04,094 DEBUG 
org.apache.flink.runtime.rest.handler.job.metrics.JobVertexMetricsHandler  - 
Received request 
/jobs/ec1c9d7a3c413a9523656efa58735009/vertices/2a751c66e0e32aee2cd8120a1a72a4d6/metrics?get=0.currentLowWatermark.
2018-02-22 13:41:04,142 DEBUG 
org.apache.flink.runtime.rest.handler.job.metrics.JobVertexMetricsHandler  - 
Received request 
/jobs/ec1c9d7a3c413a9523656efa58735009/vertices/37ecc85b429bd08d0fd539532055e117/metrics?get=0.currentLowWatermark.
2018-02-22 13:41:04,173 DEBUG 
org.apache.flink.runtime.rest.handler.job.metrics.JobVertexMetricsHandler  - 
Received request 
/jobs/ec1c9d7a3c413a9523656efa58735009/vertices/20e20298680571979f690d36d1a6db36/metrics?get=0.currentLowWatermark.
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8799) Make AbstractYarnClusterDescriptor immutable

2018-02-27 Thread Gary Yao (JIRA)

Gary Yao created FLINK-8799:
---

 Summary: Make AbstractYarnClusterDescriptor immutable
 Key: FLINK-8799
 URL: https://issues.apache.org/jira/browse/FLINK-8799
 Project: Flink
  Issue Type: Bug
  Components: YARN
Affects Versions: 1.5.0
Reporter: Gary Yao
 Fix For: 1.6.0


{{AbstractYarnClusterDescriptor}} should be made immutable. Currently, its 
internal configuration is modified from different places which makes it 
difficult to reason about the code. For example, it should not be possible to 
modify the {{zookeeperNamespace}} using a setter method. A user of this class 
should be forced to provide all information prior to creating the instance, 
e.g., by passing a {{org.apache.flink.configuration.Configuration}} object.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8798) Make commons-logging a parent-first pattern

2018-02-27 Thread Stephan Ewen (JIRA)

Stephan Ewen created FLINK-8798:
---

 Summary: Make commons-logging a parent-first pattern
 Key: FLINK-8798
 URL: https://issues.apache.org/jira/browse/FLINK-8798
 Project: Flink
  Issue Type: Bug
  Components: Core
Affects Versions: 1.4.1
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.5.0, 1.4.2, 1.6.0


The Apache {{commons-logging}} framework does not play well with child-first 
classloading.

We need to make this a parent-first pattern.

As a matter of fact, other frameworks that use inverted classloading (JBoss, 
Tomcat) use force this library to be always parent-first as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Releasing Flink 1.5.0

2018-02-27 Thread Till Rohrmann

The release branch is now created [1]. Please be aware that we should only
commit bug fixes to this branch henceforth.

@Bowen, let's wait what Aljoscha says concerning FLINK-8667. If he agrees,
then we can still merge it into the release branch.

[1]
https://git-wip-us.apache.org/repos/asf?p=flink.git;a=shortlog;h=refs/heads/release-1.5

Cheers,
Till

On Mon, Feb 26, 2018 at 8:03 PM, Bowen Li  wrote:

> Hi guys,
>
> Can we get FLINK-8667  into
> 1.5.0? I've been waiting for it for quite a few days.
>
> The reason I really want to get it into 1.5.0 is because
> KeyedBroadcastProcessFunction will be released in 1.5.0 and there's no
> compatibility issues. If we don't get FLINK-8667 in, it will be much harder
> to make it work in post-1.5.0 branches and migrate user apps.
>
> Thanks,
> Bowen
>
>
>
> On Mon, Feb 26, 2018 at 7:21 AM, Till Rohrmann 
> wrote:
>
>> +1 for cutting the release branch now.
>>
>> I'm also happy to volunteer as the release manager for Flink 1.5.
>>
>> Cheers,
>> Till
>>
>> On Mon, Feb 26, 2018 at 3:45 PM, Aljoscha Krettek 
>> wrote:
>>
>> > End of last week was the day where we wanted to to the cut of the
>> release
>> > branch. There are still a bunch of open blocker issues about bugs in our
>> > Jira: [1]. So I'm wondering whether we should actually cut the branch
>> now
>> > because some commits would have to be merged on release-1.4,
>> release-1.5,
>> > and master. What do you think?
>> >
>> > Regarding managing the release: I will have two weeks in mid march
>> where I
>> > won't have time and this could be the hot release phase. I think that
>> > because of this, it would be better to have a release manager other
>> than me
>> > and I think Till would be a good candidate for that since he's the lead
>> of
>> > FLIP-6 which is the prime new feature of the next release. What do you
>> > think about that?
>> >
>> > [1] https://issues.apache.org/jira/issues/?jql=project%20%
>> > 3D%20FLINK%20and%20priority%20%3D%20blocker%20and%
>> > 20fixVersion%20%3D%201.5.0%20and%20resolution%20%3D%20unresolved
>> >
>> > > On 13. Feb 2018, at 10:00, Till Rohrmann 
>> wrote:
>> > >
>> > > +1 from my side.
>> > >
>> > > Cheers,
>> > > Till
>> > >
>> > > On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <
>> pi...@data-artisans.com
>> > >
>> > > wrote:
>> > >
>> > >> +0.95 from my side.
>> > >>
>> > >> Network changes are mostly reviewed and should be merged by the end
>> of
>> > >> this week.
>> > >>
>> > >> Piotrek
>> > >>
>> > >>> On 12 Feb 2018, at 17:41, Stephan Ewen  wrote:
>> > >>>
>> > >>> I agree with the basic idea. I think there is no need to call it
>> "soft
>> > >>> feature freeze" though - it is a feature freeze (no new features get
>> > >>> merged) ;-)
>> > >>>
>> > >>> What you are suggesting is to delay forking of the release-1.5
>> branch
>> > to
>> > >>> avoid applying the bug fixes to too many branches. That makes sense.
>> > >>> In effect, there is a period of one week (next week) where no post
>> 1.5
>> > >>> features can be merged (feature freeze but release branch not
>> forked),
>> > >>> which should be okay.
>> > >>>
>> > >>> Best,
>> > >>> Stephan
>> > >>>
>> > >>>
>> > >>> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
>> > >> k.klou...@data-artisans.com
>> >  wrote:
>> > >>>
>> >  For me as well +1.
>> > 
>> >  Cheers,
>> >  Kostas
>> > 
>> > > On Feb 12, 2018, at 2:59 PM, Timo Walther 
>> > wrote:
>> > >
>> > > Sounds good to me. +1 from my side.
>> > >
>> > > Regards,
>> > > Timo
>> > >
>> > >
>> > > Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
>> > >> I agree with Chesnay: we should do a soft "feature freeze" first,
>> > were
>> >  we agree to not merge new features to master after that and then to
>> > the
>> >  actual hard cutting of the release branch a while later.
>> > >>
>> > >> For actual dates, I'm proposing end of this week (16.02.2018) as
>> > soft
>> >  feature freeze and end of next week (23.02.2018) as the hard cut of
>> > the
>> >  release branch?
>> > >>
>> > >> What do you think?
>> > >>
>> > >> --
>> > >> Aljoscha
>> > >>
>> > >>> On 8. Feb 2018, at 10:15, Till Rohrmann 
>> > >> wrote:
>> > >>>
>> > >>> Local state recovery is almost completely done. Only some
>> reviews
>> > and
>> > >>> merging of the final PRs is pending.
>> > >>>
>> > >>> The network stack improvements are on a good way to be finished
>> by
>> > >> the
>> >  end
>> > >>> of this week or beginning of next week. To my knowledge we got
>> > >> recently
>> > >>> green Travis builds :-) The network stack changes will also
>> include
>> > >> the
>> > >>> application level flow control and the back pressure based
>> > checkpoint
>> > >>>

Re: [DISCUS] Flink SQL Client dependency management

2018-02-27 Thread Timo Walther


Hi Xingcan,

thank you for your feedback. Regarding (3) we also thought about that 
but this approach would not scale very well. Given that we might have 
fat jars for multiple versions (Kafka 0.8, Kafka 0.6 etc.) such an 
all-in-one solution JAR file might easily go beyond 1 or 2 GB. I don't 
know if users want to download that just for a combination of connector 
and format.


Timo


Am 2/27/18 um 2:16 PM schrieb Xingcan Cui:

Hi Timo,

thanks for your efforts. Personally, I think the second option would be better 
and here are my feelings.

(1) The SQL client is designed to offer a convenient way for users to 
manipulate data with Flink. Obviously, the second option would be more 
easy-to-use.

(2) The script will help to manage the dependencies automatically, but with 
less flexibility. Once the script cannot meet the need, users have to modify it 
themselves.

(3) I wonder whether we could package all these built-in connectors and formats 
into a single JAR. With this all-in-one solution, users don’t need to consider 
much about the dependencies.

Best,
Xingcan


On 27 Feb 2018, at 6:38 PM, Stephan Ewen  wrote:

My first intuition would be to go for approach #2 for the following reasons

- I expect that in the long run, the scripts will not be that simple to
maintain. We saw that with all shell scripts thus far: they start simple,
and then grow with many special cases for this and that setup.

- Not all users have Maven, automatically downloading and configuring
Maven could be an option, but that makes the scripts yet more tricky.

- Download-and-drop-in is probably still easier to understand for users
than the syntax of a script with its parameters

- I think it may actually be even simpler to maintain for us, because all
it does is add a profile or build target to each connector to also create
the fat jar.

- Storage space is no longer really a problem. Worst case we host the fat
jars in an S3 bucket.


On Mon, Feb 26, 2018 at 7:33 PM, Timo Walther  wrote:


Hi everyone,

as you may know a first minimum version of FLIP-24 [1] for the upcoming
Flink SQL Client has been merged to the master. We also merged
possibilities to discover and configure table sources without a single line
of code using string-based properties [2] and Java service provider
discovery.

We are now facing the issue of how to manage dependencies in this new
environment. It is different from how regular Flink projects are created
(by setting up a a new Maven project and build a jar or fat jar). Ideally,
a user should be able to select from a set of prepared connectors,
catalogs, and formats. E.g., if a Kafka connector and Avro format is
needed, all that should be required is to move a "flink-kafka.jar" and
"flink-avro.jar" into the "sql_lib" directory that is shipped to a Flink
cluster together with the SQL query.

The question is how do we want to offer those JAR files in the future? We
see two options:

1) We prepare Maven build profiles for all offered modules and provide a
shell script for building fat jars. A script call could look like
"./sql-client-dependency.sh kafka 0.10". It would automatically download
what is needed and place the JAR file in the library folder. This approach
would keep our development effort low but would require Maven to be present
and builds to pass on different environments (e.g. Windows).

2) We build fat jars for these modules with every Flink release that can
be hostet somewhere (e.g. Apache infrastructure, but not Maven central).
This would make it very easy to add a dependency by downloading the
prepared JAR files. However, it would require to build and host large fat
jars for every connector (and version) with every Flink major and minor
release. The size of such a repository might grow quickly.

What do you think? Do you see other options to make adding dependencies as
possible?


Regards,

Timo


[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client

[2] https://issues.apache.org/jira/browse/FLINK-8240

Re: Checkpointing Event Time Watermarks

2018-02-27 Thread Xingcan Cui

Hi Vijay,

normally, maybe there’s no need to checkpoint the event times / watermarks 
since they are automatically generated based on the records. What’s your 
intention?

Best,
Xingcan

> On 27 Feb 2018, at 8:50 PM, vijay kansal  wrote:
> 
> Hi All
> 
> Is there a way to checkpoint event time watermarks in Flink ?
> 
> I tries searching for this, but could not figure out...
> 
> 
> Vijay Kansal
> Software Development Engineer
> LimeRoad

Re: [DISCUS] Flink SQL Client dependency management

2018-02-27 Thread Xingcan Cui

Hi Timo,

thanks for your efforts. Personally, I think the second option would be better 
and here are my feelings. 

(1) The SQL client is designed to offer a convenient way for users to 
manipulate data with Flink. Obviously, the second option would be more 
easy-to-use. 

(2) The script will help to manage the dependencies automatically, but with 
less flexibility. Once the script cannot meet the need, users have to modify it 
themselves. 

(3) I wonder whether we could package all these built-in connectors and formats 
into a single JAR. With this all-in-one solution, users don’t need to consider 
much about the dependencies.

Best,
Xingcan

> On 27 Feb 2018, at 6:38 PM, Stephan Ewen  wrote:
> 
> My first intuition would be to go for approach #2 for the following reasons
> 
> - I expect that in the long run, the scripts will not be that simple to
> maintain. We saw that with all shell scripts thus far: they start simple,
> and then grow with many special cases for this and that setup.
> 
> - Not all users have Maven, automatically downloading and configuring
> Maven could be an option, but that makes the scripts yet more tricky.
> 
> - Download-and-drop-in is probably still easier to understand for users
> than the syntax of a script with its parameters
> 
> - I think it may actually be even simpler to maintain for us, because all
> it does is add a profile or build target to each connector to also create
> the fat jar.
> 
> - Storage space is no longer really a problem. Worst case we host the fat
> jars in an S3 bucket.
> 
> 
> On Mon, Feb 26, 2018 at 7:33 PM, Timo Walther  wrote:
> 
>> Hi everyone,
>> 
>> as you may know a first minimum version of FLIP-24 [1] for the upcoming
>> Flink SQL Client has been merged to the master. We also merged
>> possibilities to discover and configure table sources without a single line
>> of code using string-based properties [2] and Java service provider
>> discovery.
>> 
>> We are now facing the issue of how to manage dependencies in this new
>> environment. It is different from how regular Flink projects are created
>> (by setting up a a new Maven project and build a jar or fat jar). Ideally,
>> a user should be able to select from a set of prepared connectors,
>> catalogs, and formats. E.g., if a Kafka connector and Avro format is
>> needed, all that should be required is to move a "flink-kafka.jar" and
>> "flink-avro.jar" into the "sql_lib" directory that is shipped to a Flink
>> cluster together with the SQL query.
>> 
>> The question is how do we want to offer those JAR files in the future? We
>> see two options:
>> 
>> 1) We prepare Maven build profiles for all offered modules and provide a
>> shell script for building fat jars. A script call could look like
>> "./sql-client-dependency.sh kafka 0.10". It would automatically download
>> what is needed and place the JAR file in the library folder. This approach
>> would keep our development effort low but would require Maven to be present
>> and builds to pass on different environments (e.g. Windows).
>> 
>> 2) We build fat jars for these modules with every Flink release that can
>> be hostet somewhere (e.g. Apache infrastructure, but not Maven central).
>> This would make it very easy to add a dependency by downloading the
>> prepared JAR files. However, it would require to build and host large fat
>> jars for every connector (and version) with every Flink major and minor
>> release. The size of such a repository might grow quickly.
>> 
>> What do you think? Do you see other options to make adding dependencies as
>> possible?
>> 
>> 
>> Regards,
>> 
>> Timo
>> 
>> 
>> [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client
>> 
>> [2] https://issues.apache.org/jira/browse/FLINK-8240
>> 
>>

Checkpointing Event Time Watermarks

2018-02-27 Thread vijay kansal

Hi All

Is there a way to checkpoint event time watermarks in Flink ?

I tries searching for this, but could not figure out...


Vijay Kansal
Software Development Engineer
LimeRoad

[jira] [Created] (FLINK-8797) Port AbstractOperatorRestoreTestBase to MiniClusterResource

2018-02-27 Thread Aljoscha Krettek (JIRA)

Aljoscha Krettek created FLINK-8797:
---

 Summary: Port AbstractOperatorRestoreTestBase to 
MiniClusterResource
 Key: FLINK-8797
 URL: https://issues.apache.org/jira/browse/FLINK-8797
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
 Fix For: 1.5.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release 1.4.2, release candidate #2

2018-02-27 Thread Tzu-Li (Gordon) Tai

Sorry for the incorrect vote thread title.
This vote is for release candidate #1, not 2.

Cheers,
Gordon

On 27 February 2018 at 8:30:42 PM, Tzu-Li (Gordon) Tai (tzuli...@apache.org) 
wrote:

Hi everyone,

Please review and vote on release candidate #1 for Flink 1.4.2, as follows: 
[ ] +1, Approve the release 
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes: 
* JIRA release notes [1], 
* the official Apache source release and binary convenience releases to be 
deployed to dist.apache.org [2], which are signed with the key with fingerprint 
1C1E2394D3194E1944613488F320986D35C33D6A [3], 
* all artifacts to be deployed to the Maven Central Repository [4], 
* source code tag “release-1.4.2-rc1” [5], 
* website pull request listing the new release [6]. 
* A complete list of all new commits in release-1.4.2-rc1, since release-1.4.1 
[7]

The vote will be open for at least 72 hours.
Please test the release and vote for the release candidate before Friday (March 
2nd), 7pm CET.
It is adopted by majority approval, with at least 3 PMC affirmative votes.

Thanks, 
Gordon

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12342745
[2] http://people.apache.org/~tzulitai/flink-1.4.2-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1148
[5] 
https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=d5ef96aeb63ad3dd4d0676e0c32a35e888ecb409
[6] https://github.com/apache/flink-web/pull/102
[7]
* b74c705157 - [FLINK-8543] Don’t call super.close() in AvroKeyValueSinkWriter 
(16 hours ago)
* a0193f10af - [FLINK-8772] [kafka] Fix missing log parameter (21 hours ago)
* 0396fc8c03 - [hotfix] [test] Also trap INT signal in Kafka end-to-end test 
(21 hours ago)
* 59169d0eeb - [hotfix] [test] Make test-streaming-kafka010.sh more flexible 
for local execution (21 hours ago)
* cb6a8a68c4 - [FLINK-8741] [kafka] Fix incorrect user code classloader in 
FlinkKafkaConsumer (21 hours ago)
* 0d89a1c9f1 - [hotfix] Update docs version to Flink 1.4.1 (4 days ago)
* f06ec38f22 - [FLINK-8735] Add new StatefulJobSavepointMigrationITCase (5 days 
ago)
* 82e6f8d5b8 - [FLINK-8735] Rename StatefulJobSavepointMigrationITCase (6 days 
ago)
* a7df42485a - [FLINK-8574][travis] Add timestamp to logging messages (7 days 
ago)
* 527faf63e2 - [hotfix][prometheus][tests] Add utility for generating port 
ranges (7 days ago)
* d88f43bb06 - [hotfix][prometheus] Document internal usage of 
CollectorRegistry.defaultRegistry (7 days ago)
* 45efe4702a - [FLINK-8621][prometheus][tests] Remove 
endpointIsUnavailableAfterReporterIsClosed() (7 days ago)
* 528317c8f2 - [FLINK-8692][docs] Remove extra parenthesis in scala code 
samples (9 days ago)
* cc76c323f0 - [FLINK-8303] [docs] Allow to overwrite ruby/gem binary (11 days 
ago)
* d4435e1596 - [FLINK-8303] Add hawkins back to Gemfile (11 days ago)
* 50b648473f - [FLINK-8308] Remove explicit yajl-ruby dependency, update Jekyll 
to 3+ (11 days ago)
* 054af99746 - [FLINK-8520][cassandra] Fix race condition (11 days ago)
* 1b70f50d93 - [FLINK-8576][QS] Reduce verbosity when classes can’t be found 
(11 days ago)
* f2b5635bc8 - [hotfix] Remove more checkState() calls from SharedBuffer 
serialization (12 days ago)
* 59f9ded8fd - [FLINK-8652] [QS] Reduce log level in getKvState to DEBUG. (13 
days ago)
* a044d9d6ca - [hotfix] Remove costly logging statements from CEP SharedBuffer 
(13 days ago)
* bafb91eeb5 - [FLINK-8423] OperatorChain#pushToOperator catch block may fail 
with NPE (2 weeks ago)

[VOTE] Release 1.4.2, release candidate #2

2018-02-27 Thread Tzu-Li (Gordon) Tai

Hi everyone,

Please review and vote on release candidate #1 for Flink 1.4.2, as follows: 
[ ] +1, Approve the release 
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes: 
* JIRA release notes [1], 
* the official Apache source release and binary convenience releases to be 
deployed to dist.apache.org [2], which are signed with the key with fingerprint 
1C1E2394D3194E1944613488F320986D35C33D6A [3], 
* all artifacts to be deployed to the Maven Central Repository [4], 
* source code tag “release-1.4.2-rc1” [5], 
* website pull request listing the new release [6]. 
* A complete list of all new commits in release-1.4.2-rc1, since release-1.4.1 
[7]

The vote will be open for at least 72 hours.
Please test the release and vote for the release candidate before Friday (March 
2nd), 7pm CET.
It is adopted by majority approval, with at least 3 PMC affirmative votes.

Thanks, 
Gordon

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12342745
[2] http://people.apache.org/~tzulitai/flink-1.4.2-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1148
[5] 
https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=d5ef96aeb63ad3dd4d0676e0c32a35e888ecb409
[6] https://github.com/apache/flink-web/pull/102
[7]
* b74c705157 - [FLINK-8543] Don’t call super.close() in AvroKeyValueSinkWriter 
(16 hours ago)
* a0193f10af - [FLINK-8772] [kafka] Fix missing log parameter (21 hours ago)
* 0396fc8c03 - [hotfix] [test] Also trap INT signal in Kafka end-to-end test 
(21 hours ago)
* 59169d0eeb - [hotfix] [test] Make test-streaming-kafka010.sh more flexible 
for local execution (21 hours ago)
* cb6a8a68c4 - [FLINK-8741] [kafka] Fix incorrect user code classloader in 
FlinkKafkaConsumer (21 hours ago)
* 0d89a1c9f1 - [hotfix] Update docs version to Flink 1.4.1 (4 days ago)
* f06ec38f22 - [FLINK-8735] Add new StatefulJobSavepointMigrationITCase (5 days 
ago)
* 82e6f8d5b8 - [FLINK-8735] Rename StatefulJobSavepointMigrationITCase (6 days 
ago)
* a7df42485a - [FLINK-8574][travis] Add timestamp to logging messages (7 days 
ago)
* 527faf63e2 - [hotfix][prometheus][tests] Add utility for generating port 
ranges (7 days ago)
* d88f43bb06 - [hotfix][prometheus] Document internal usage of 
CollectorRegistry.defaultRegistry (7 days ago)
* 45efe4702a - [FLINK-8621][prometheus][tests] Remove 
endpointIsUnavailableAfterReporterIsClosed() (7 days ago)
* 528317c8f2 - [FLINK-8692][docs] Remove extra parenthesis in scala code 
samples (9 days ago)
* cc76c323f0 - [FLINK-8303] [docs] Allow to overwrite ruby/gem binary (11 days 
ago)
* d4435e1596 - [FLINK-8303] Add hawkins back to Gemfile (11 days ago)
* 50b648473f - [FLINK-8308] Remove explicit yajl-ruby dependency, update Jekyll 
to 3+ (11 days ago)
* 054af99746 - [FLINK-8520][cassandra] Fix race condition (11 days ago)
* 1b70f50d93 - [FLINK-8576][QS] Reduce verbosity when classes can’t be found 
(11 days ago)
* f2b5635bc8 - [hotfix] Remove more checkState() calls from SharedBuffer 
serialization (12 days ago)
* 59f9ded8fd - [FLINK-8652] [QS] Reduce log level in getKvState to DEBUG. (13 
days ago)
* a044d9d6ca - [hotfix] Remove costly logging statements from CEP SharedBuffer 
(13 days ago)
* bafb91eeb5 - [FLINK-8423] OperatorChain#pushToOperator catch block may fail 
with NPE (2 weeks ago)

[jira] [Created] (FLINK-8796) Update "Upgrading Applications and Flink Versions" for 1.5

2018-02-27 Thread Till Rohrmann (JIRA)

Till Rohrmann created FLINK-8796:


 Summary: Update "Upgrading Applications and Flink Versions" for 1.5
 Key: FLINK-8796
 URL: https://issues.apache.org/jira/browse/FLINK-8796
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.5.0
Reporter: Till Rohrmann
 Fix For: 1.5.0


Update the Flink documentation for upgrading Flink applications from a previous 
version to 1.5.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUS] Flink SQL Client dependency management

2018-02-27 Thread Stephan Ewen

My first intuition would be to go for approach #2 for the following reasons

  - I expect that in the long run, the scripts will not be that simple to
maintain. We saw that with all shell scripts thus far: they start simple,
and then grow with many special cases for this and that setup.

  - Not all users have Maven, automatically downloading and configuring
Maven could be an option, but that makes the scripts yet more tricky.

  - Download-and-drop-in is probably still easier to understand for users
than the syntax of a script with its parameters

  - I think it may actually be even simpler to maintain for us, because all
it does is add a profile or build target to each connector to also create
the fat jar.

  - Storage space is no longer really a problem. Worst case we host the fat
jars in an S3 bucket.



On Mon, Feb 26, 2018 at 7:33 PM, Timo Walther  wrote:

> Hi everyone,
>
> as you may know a first minimum version of FLIP-24 [1] for the upcoming
> Flink SQL Client has been merged to the master. We also merged
> possibilities to discover and configure table sources without a single line
> of code using string-based properties [2] and Java service provider
> discovery.
>
> We are now facing the issue of how to manage dependencies in this new
> environment. It is different from how regular Flink projects are created
> (by setting up a a new Maven project and build a jar or fat jar). Ideally,
> a user should be able to select from a set of prepared connectors,
> catalogs, and formats. E.g., if a Kafka connector and Avro format is
> needed, all that should be required is to move a "flink-kafka.jar" and
> "flink-avro.jar" into the "sql_lib" directory that is shipped to a Flink
> cluster together with the SQL query.
>
> The question is how do we want to offer those JAR files in the future? We
> see two options:
>
> 1) We prepare Maven build profiles for all offered modules and provide a
> shell script for building fat jars. A script call could look like
> "./sql-client-dependency.sh kafka 0.10". It would automatically download
> what is needed and place the JAR file in the library folder. This approach
> would keep our development effort low but would require Maven to be present
> and builds to pass on different environments (e.g. Windows).
>
> 2) We build fat jars for these modules with every Flink release that can
> be hostet somewhere (e.g. Apache infrastructure, but not Maven central).
> This would make it very easy to add a dependency by downloading the
> prepared JAR files. However, it would require to build and host large fat
> jars for every connector (and version) with every Flink major and minor
> release. The size of such a repository might grow quickly.
>
> What do you think? Do you see other options to make adding dependencies as
> possible?
>
>
> Regards,
>
> Timo
>
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client
>
> [2] https://issues.apache.org/jira/browse/FLINK-8240
>
>

Re: [DISCUSS] Release Flink 1.4.2

2018-02-27 Thread Stephan Ewen

+1 to an immediate 1.4.2 release


On Tue, Feb 27, 2018 at 9:43 AM, Ufuk Celebi  wrote:

> Since users already reported this on the ML, I'm big +1 to release
> immediately.
>
> On Tue, Feb 27, 2018 at 7:45 AM, Tzu-Li (Gordon) Tai
>  wrote:
> > Since we’re still a bit undecided on FLINK-8451, I think it would be
> best to release 1.4.2 with FLINK-8741 now.
> > Otherwise, Kafka connectors using custom watermark assigners are
> basically unusable right now.
> >
> > I’ll start the build for RC1 for 1.4.2, and will start the vote by the
> end of the day.
> > If you object and think we should still try to include FLINK-8451,
> please let me know.
> >
> > Cheers,
> > Gordon
> >
> > On 23 February 2018 at 8:27:38 PM, Stephan Ewen (se...@apache.org)
> wrote:
> >
> > How about releasing 1.4.2 now, meaning immediately. This can be very
> > lightweight.
> >
> > FLINK-8451 looks like it should have more thorough testing and should go
> > into 1.4.3.
> > I think there is no harm in more frequent bugfix releases.
> >
> >
> > On Fri, Feb 23, 2018 at 9:56 AM, Timo Walther 
> wrote:
> >
> >> I also almost have a fix ready for FLINK-8451. I think it should also go
> >> into 1.4.2.
> >>
> >> Regards,
> >> Timo
> >>
> >>
> >> Am 2/22/18 um 11:29 AM schrieb Aljoscha Krettek:
> >>
> >> They reason they didn't catch this is that the bug only occurs if users
> >>> use a custom timestamp/watermark assigner. But yes, we should be able
> to
> >>> extend the end-to-end tests to catch this.
> >>>
> >>> On 22. Feb 2018, at 11:05, Till Rohrmann  wrote:
> 
>  If the Kafka connectors are unusable with 1.4.1, then I would be in
> favor
>  of releasing 1.4.2 asap.
> 
>  I'm wondering why the end-to-end Kafka tests did not catch this
> problem.
>  Maybe we could adapt them such that they guard against it in the
> future.
> 
>  Cheers,
>  Till
> 
>  On Thu, Feb 22, 2018 at 9:46 AM, Tzu-Li (Gordon) Tai <
>  tzuli...@apache.org>
>  wrote:
> 
>  Hi all,
> >
> > Unfortunately, we've discovered a bug in 1.4.1, which suggests that
> we
> > should almost immediately release another bugfix release:
> > https://issues.apache.org/jira/browse/FLINK-8741.
> >
> > Since this issue was introduced only in 1.4.1, it might make sense to
> > release 1.4.2 with only the fix for FLINK-8741 included. What do you
> > think?
> >
> > Best,
> > Gordon
> >
> >  > source=link_campaign=sig-email_content=webmail>
> > 不含病毒。www.avast.com
> >  > source=link_campaign=sig-email_content=webmail>
> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >
> >
> >>
>

[jira] [Created] (FLINK-8795) Failed to submit JobGraph Apache Flink on Github Master Branch

2018-02-27 Thread kant kodali (JIRA)

kant kodali created FLINK-8795:
--

 Summary: Failed to submit JobGraph Apache Flink on Github Master 
Branch
 Key: FLINK-8795
 URL: https://issues.apache.org/jira/browse/FLINK-8795
 Project: Flink
  Issue Type: Bug
Reporter: kant kodali


I am trying to run the simple code below after building everything from Flink's 
github master branch for various reasons. I get an exception below and I wonder 
what runs on port 9065? and How to fix this exception?

I followed the instructions from the Flink master branch so I did the following.

 

{{git clone https://github.com/apache/flink.git }}
{{cd flink mvn clean package -DskipTests }}
{{cd build-target }}
{{./bin/start-scala-shell.sh local}}

{{And Here is the code I ran}}

 
{code:java}
val dataStream = senv.fromElements(1, 2, 3, 4)
dataStream.countWindowAll(2).sum(0).print()
senv.execute("My streaming program"){code}
{{And I finally get this exception}}

{{}}
{code:java}
Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to 
submit JobGraph. at 
org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$18(RestClusterClient.java:306)
 at 
java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
 at 
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
 at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) 
at 
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
 at 
org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$222(RestClient.java:196)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:268)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:284)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
 at java.lang.Thread.run(Thread.java:745) Caused by: 
java.util.concurrent.CompletionException: java.net.ConnectException: Connection 
refused: localhost/127.0.0.1:9065 at 
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
 at 
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
 at 
java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943) 
at 
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
 ... 16 more Caused by: java.net.ConnectException: Connection refused: 
localhost/127.0.0.1:9065 at sun.nio.ch.SocketChannelImpl.checkConnect(Native 
Method) at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at 
org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:281){code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

2018-02-27 Thread yanxiaobin (JIRA)

yanxiaobin created FLINK-8794:
-

 Summary: When using BucketingSink, it happens that one of the 
files is always in the [.in-progress] state
 Key: FLINK-8794
 URL: https://issues.apache.org/jira/browse/FLINK-8794
 Project: Flink
  Issue Type: Bug
  Components: filesystem-connector
Affects Versions: 1.4.0, 1.4.1
Reporter: yanxiaobin


When using BucketingSink, it happens that one of the files is always in the 
[.in-progress] state.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: How to use Dimension table in Flink TableAPI with StreamExecutionEnvironment ?

2018-02-27 Thread Fabian Hueske

Hi,

Liu and Hequn are right.
You need to pass at least one parameter into the table function, i.e.,

select t.col_1 from test t left join lateral
table(dim_test(SOME_ATTRIBUTE)) b on true

Best, Fabian

2018-02-24 13:24 GMT+01:00 ZhenBao Ye :

> hi，i was use 1.4.0。
>
> Yezhenbao
>
> > 在 2018年2月24日，17:55，Renjie Liu  写道：
> >
> > Hi, according to flink doc, it seems that you need to pass at least one
> > argument into the table function.
> >
> >> On Fri, Feb 23, 2018 at 12:35 AM 叶振宝 <827295...@qq.com> wrote:
> >>
> >> Hey, I am new to flink and I have a question and want to see if anyone
> can
> >> help here.
> >>
> >> How to use Dimension table in Flink TableAPI with
> >> StreamExecutionEnvironment ?
> >>
> >> I use TableFuncion to deal this question, but it have some problem in
> debug
> >> like this:
> >> LogicalProject(col_1=[$0])
> >>  LogicalJoin(condition=[true], joinType=[left])
> >>LogicalTableScan(table=[[test]])
> >>LogicalTableFunctionScan(invocation=[dim_test()],
> >> rowType=[RecordType(VARCHAR(65536) col, VARCHAR(65536) name)],
> >> elementType=[class [Ljava.lang.Object;])
> >>
> >> This exception indicates that the query uses an unsupported SQL feature.
> >> Please check the documentation for the set of currently supported SQL
> >> features.
> >>at
> >> org.apache.flink.table.api.TableEnvironment.runVolcanoPlanner(
> TableEnvironment.scala:274)
> >>at
> >> org.apache.flink.table.api.StreamTableEnvironment.optimize(
> StreamTableEnvironment.scala:674)
> >>at
> >> org.apache.flink.table.api.StreamTableEnvironment.translate(
> StreamTableEnvironment.scala:730)
> >>at
> >> org.apache.flink.table.api.StreamTableEnvironment.writeToSink(
> StreamTableEnvironment.scala:216)
> >>at
> >> org.apache.flink.table.api.TableEnvironment.insertInto(
> TableEnvironment.scala:692)
> >>at
> >> com.bigdata.stream.streamsql.FlinkSqlTest.main(FlinkSqlTest.java:64)
> >>
> >> SQL : select t.col_1 from test t left join lateral table(dim_test()) b
> on
> >> true
> >>
> >> Main Code:
> >> public static void main(String[] args) throws Exception {
> >>String sql = "select t.col_1 from test t left join lateral
> >> table(dim_test()) b on true";
> >>StreamExecutionEnvironment streamEnv =
> >> StreamExecutionEnvironment.getExecutionEnvironment();
> >>StreamTableEnvironment stEnv =
> >> TableEnvironment.getTableEnvironment(streamEnv);
> >>Properties kafkaProps = new Properties();
> >>kafkaProps.setProperty("zookeeper.connect", "localhost:2181");
> >>kafkaProps.setProperty("bootstrap.servers", "localhost:9092");
> >>kafkaProps.setProperty("group.id", "test");
> >>Kafka010JsonTableSource tableSource =
> >> Kafka010JsonTableSource.builder()
> >>.forTopic("test")
> >>.withKafkaProperties(kafkaProps)
> >>.withSchema(TableSchema.builder()
> >>.field("col_1", Types.STRING)
> >>.field("col_2",Types.STRING).build())
> >>.build();
> >>stEnv.registerTableSource("test", tableSource);
> >>String[] columns = {"col","name"};
> >>TypeInformation[] typeInformations =
> >> {TypeInformation.of(String.class),TypeInformation.of(String.class)};
> >>TableSchema tableSchema = new
> >> TableSchema(columns,typeInformations);
> >>Map context = new HashMap<>();
> >>context.put("mysql.url","jdbc:mysql://localhost:3306/test");
> >>context.put("mysql.driver","com.mysql.jdbc.Driver");
> >>context.put("mysql.user","test");
> >>context.put("mysql.password","test");
> >>context.put("mysql.table","dim_test");
> >>StreamSqlDim dim = new
> >> MySqlDimFactory().getInstance(tableSchema,new
> StreamSqlContext(context));
> >>stEnv.registerFunction("dim_test",dim);
> >>
> >>String[] outColumns = {"col"};
> >>TypeInformation[] outType = {TypeInformation.of(String.class)};
> >>TableSink tableSink = new
> >> Kafka010JsonTableSink("test_out",kafkaProps);
> >>stEnv.registerTableSink("test_out",outColumns,outType,
> tableSink);
> >>Table t = stEnv.sql(sql);
> >>stEnv.insertInto(t,"test_out",stEnv.queryConfig());
> >>streamEnv.execute();
> >>}
> >>
> >> MySqlDim is extends TableFunction ,and the method eval() is empty,like
> >> this:
> >> public void eval(){
> >>
> >> }
> >>
> >>
> >>
> >> --
> > Liu, Renjie
> > Software Engineer, MVAD
> >
>
>
>
>

[jira] [Created] (FLINK-8793) Only key with password are hidden in Flink web interface

2018-02-27 Thread Etienne CARRIERE (JIRA)

Etienne CARRIERE created FLINK-8793:
---

 Summary: Only key with password are hidden in Flink web interface
 Key: FLINK-8793
 URL: https://issues.apache.org/jira/browse/FLINK-8793
 Project: Flink
  Issue Type: Bug
  Components: REST
Affects Versions: 1.4.1
Reporter: Etienne CARRIERE
 Fix For: 1.5.0


Currently, we going in /jobmanager/config on the web interface, the value of 
the key containing "password" are replaced by "" 

When using s3 for checkpoint/savepoint configuration on an infrastructure which 
is not on AWS (where IAM is not possible), the s3.secret-key is revealed from 
the interface. 

I propose the same behaviour as key with "password" for key with "secret" 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Release Flink 1.4.2

2018-02-27 Thread Ufuk Celebi

Since users already reported this on the ML, I'm big +1 to release immediately.

On Tue, Feb 27, 2018 at 7:45 AM, Tzu-Li (Gordon) Tai
 wrote:
> Since we’re still a bit undecided on FLINK-8451, I think it would be best to 
> release 1.4.2 with FLINK-8741 now.
> Otherwise, Kafka connectors using custom watermark assigners are basically 
> unusable right now.
>
> I’ll start the build for RC1 for 1.4.2, and will start the vote by the end of 
> the day.
> If you object and think we should still try to include FLINK-8451, please let 
> me know.
>
> Cheers,
> Gordon
>
> On 23 February 2018 at 8:27:38 PM, Stephan Ewen (se...@apache.org) wrote:
>
> How about releasing 1.4.2 now, meaning immediately. This can be very
> lightweight.
>
> FLINK-8451 looks like it should have more thorough testing and should go
> into 1.4.3.
> I think there is no harm in more frequent bugfix releases.
>
>
> On Fri, Feb 23, 2018 at 9:56 AM, Timo Walther  wrote:
>
>> I also almost have a fix ready for FLINK-8451. I think it should also go
>> into 1.4.2.
>>
>> Regards,
>> Timo
>>
>>
>> Am 2/22/18 um 11:29 AM schrieb Aljoscha Krettek:
>>
>> They reason they didn't catch this is that the bug only occurs if users
>>> use a custom timestamp/watermark assigner. But yes, we should be able to
>>> extend the end-to-end tests to catch this.
>>>
>>> On 22. Feb 2018, at 11:05, Till Rohrmann  wrote:

 If the Kafka connectors are unusable with 1.4.1, then I would be in favor
 of releasing 1.4.2 asap.

 I'm wondering why the end-to-end Kafka tests did not catch this problem.
 Maybe we could adapt them such that they guard against it in the future.

 Cheers,
 Till

 On Thu, Feb 22, 2018 at 9:46 AM, Tzu-Li (Gordon) Tai <
 tzuli...@apache.org>
 wrote:

 Hi all,
>
> Unfortunately, we've discovered a bug in 1.4.1, which suggests that we
> should almost immediately release another bugfix release:
> https://issues.apache.org/jira/browse/FLINK-8741.
>
> Since this issue was introduced only in 1.4.1, it might make sense to
> release 1.4.2 with only the fix for FLINK-8741 included. What do you
> think?
>
> Best,
> Gordon
>
>  source=link_campaign=sig-email_content=webmail>
> 不含病毒。www.avast.com
>  source=link_campaign=sig-email_content=webmail>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
>
>>

Re: [VOTE] Release 1.4.2, release candidate #2

Re: Verifying watermarks in integration test

Re: [DISCUS] Flink SQL Client dependency management

[jira] [Created] (FLINK-8801) S3's eventual consistent read-after-write may fail yarn deployment of resources to S3

[jira] [Created] (FLINK-8800) Set Logging to TRACE for org.apache.flink.runtime.rest.handler.job.metrics.JobVertexMetricsHandler

[jira] [Created] (FLINK-8799) Make AbstractYarnClusterDescriptor immutable

[jira] [Created] (FLINK-8798) Make commons-logging a parent-first pattern

Re: [DISCUSS] Releasing Flink 1.5.0

Re: [DISCUS] Flink SQL Client dependency management

Re: Checkpointing Event Time Watermarks

Re: [DISCUS] Flink SQL Client dependency management

Checkpointing Event Time Watermarks

[jira] [Created] (FLINK-8797) Port AbstractOperatorRestoreTestBase to MiniClusterResource

Re: [VOTE] Release 1.4.2, release candidate #2

[VOTE] Release 1.4.2, release candidate #2

[jira] [Created] (FLINK-8796) Update "Upgrading Applications and Flink Versions" for 1.5

Re: [DISCUS] Flink SQL Client dependency management

Re: [DISCUSS] Release Flink 1.4.2

[jira] [Created] (FLINK-8795) Failed to submit JobGraph Apache Flink on Github Master Branch

[jira] [Created] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

Re: How to use Dimension table in Flink TableAPI with StreamExecutionEnvironment ?

[jira] [Created] (FLINK-8793) Only key with password are hidden in Flink web interface

Re: [DISCUSS] Release Flink 1.4.2

23 matches

Site Navigation

Mail list logo

Footer information