[jira] [Created] (FLINK-6387) Flink UI support access log

[shijinkui (JIRA)]

shijinkui created FLINK-6387:


 Summary: Flink UI support access log
 Key: FLINK-6387
 URL: https://issues.apache.org/jira/browse/FLINK-6387
 Project: Flink
  Issue Type: Improvement
  Components: Webfrontend
Reporter: shijinkui
Assignee: shijinkui


Record the use request to the access log. Append use access to the log file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6367) support custom header settings of allow origin

[shijinkui (JIRA)]

shijinkui created FLINK-6367:


 Summary: support custom header settings of allow origin
 Key: FLINK-6367
 URL: https://issues.apache.org/jira/browse/FLINK-6367
 Project: Flink
  Issue Type: Sub-task
  Components: Webfrontend
Reporter: shijinkui


`jobmanager.web.access-control-allow-origin`: Enable custom access control 
parameter for allow origin header, default is `*`.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Java library for Flink-Kudu integration

[shijinkui]

Hi, Fabian
Do we have plan to replace Flink connectors with bahir-flink[1]?

[1] https://github.com/apache/Bahir-flink

在 2017/3/28 上午12:15， "Fabian Hueske"  写入:

>Hi Ruben,
>
>thanks for sharing this!
>A Flink Kudu connector is a great contribution and Bahir seems to be the
>right place for it.
>
>Thanks, Fabian
>
>
>2017-03-27 15:35 GMT+02:00 :
>
>> Hi all,
>>
>> I apologize for sending the email to both accounts, but not sure where
>> this topic fits better.
>>
>> In my team, we have been working in some PoCs and PoVs about new data
>> architectures. As part of this work, we have implemented a library to
>> connect Kudu and Flink. The library allows reading/writing from/to Kudu
>> tablets using DataSet API and also writing to Kudu using DataStream API.
>>
>> You can find the code and documentation (including some examples) in
>> https://github.com/rubencasado/Flink-Kudu
>>
>> Any comment/suggestion/contribution is very welcomed ☺
>>
>> We will try to publish this contribution to the Apache Bahir project.
>>
>> Best
>>
>> 
>> Rubén Casado Tejedor, PhD
>> > accenture digital
>> Big Data Manager
>> ' + 34 629 009 429
>> • ruben.casado.teje...@accenture.com> casado.teje...@accenture.com>
>>
>> 
>>
>> This message is for the designated recipient only and may contain
>> privileged, proprietary, or otherwise confidential information. If you
>>have
>> received it in error, please notify the sender immediately and delete
>>the
>> original. Any other use of the e-mail by you is prohibited. Where
>>allowed
>> by local law, electronic communications with Accenture and its
>>affiliates,
>> including e-mail and instant messaging (including content), may be
>>scanned
>> by our systems for the purposes of information security and assessment
>>of
>> internal compliance with Accenture policy.
>> 
>> __
>>
>> www.accenture.com
>>

[jira] [Created] (FLINK-6201) move python example files to the examples dir

[shijinkui (JIRA)]

shijinkui created FLINK-6201:


 Summary: move python example files to the examples dir
 Key: FLINK-6201
 URL: https://issues.apache.org/jira/browse/FLINK-6201
 Project: Flink
  Issue Type: Sub-task
  Components: Examples
Reporter: shijinkui


Python example in the resource dir is not suitable. Move them to the 
examples/python dir.
```


../flink-libraries/flink-python/src/main/python/org/apache/flink/python/api
resources/python
0755

```



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6193) Flink dist directory normalize

[shijinkui (JIRA)]

shijinkui created FLINK-6193:


 Summary: Flink dist directory normalize
 Key: FLINK-6193
 URL: https://issues.apache.org/jira/browse/FLINK-6193
 Project: Flink
  Issue Type: Improvement
  Components: Examples
Reporter: shijinkui


The Flink distribution's directory have no very clear responsibility about what 
type of files should be in which directory. For example, "opt" directories are 
mixed with library jars and example jars.

This mail here: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-dist-directory-management-td16784.html

After discuss, we determine for the distribution directory style below:
- "examples" directory only contain example jars
- "opt" directory only contain optional library jars in runtime
- "lib" directory only contain library jar that must be loaded at runtime
- "resources" directory only contain resource file used at runtime, such as web 
file



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] Flink dist directory management

[shijinkui]

Hi, Fabian, Jark Wu, Greg
Thank for your feedback and support. I’ll create some sub-issue to
complete the every task of code refactor.

在 2017/3/27 下午3:38， "Fabian Hueske" <fhue...@gmail.com> 写入:

>Hi Jinkui,
>
>I agree with you, moving all examples into ./examples would be nice.
>Note, that for the DataSet and DataStream APIs, each example is packaged
>as
>a separate JAR with the main class being included in the MANIFEST file.
>
>I think we should select a few representative examples of the Table and
>Gelly examples and do the same.
>IMO, the Table API / SQL examples could also be improved a lot, but that's
>a different story.
>
>Best, Fabian
>
>2017-03-27 5:47 GMT+02:00 伍翀(云邪) <wuchong...@alibaba-inc.com>:
>
>> Hi Jinkui,
>>
>> Thanks for bringing up this discussion. I’m +1 to moving table-examples
>> and belly-examples into "examples/“ to keep “opt” clean.
>>
>> Best,
>> Jark Wu
>>
>> > 在 2017年3月25日，下午3:46，shijinkui <shijin...@huawei.com> 写道：
>> >
>> > Hi, all
>> >
>> > The Flink distributionĄ¯s directory have no very clear responsibility
>> about what type of files should be in which directory.
>> >
>> > The "bin","conf","lib" directories are clear for their responsibility.
>> >
>> > But the Ą°opt" directories are mixed with library jars and example
>>jars.
>> >
>> > I think we can discuss how is reasonable for the directory. Once we
>> determined, we should follow it.
>> >
>> > IMO, directory style below is reasonable:
>> >
>> > - "examples" directory only contain example jars
>> > - "opt" directory only contain optional library jars in runtime
>> > - "lib" directory only contain library jar that must be loaded at
>>runtime
>> > - Ą°resourcesĄą directory only contain resource file used at runtime,
>> such as web file
>> >
>> > Show your opinion please.
>> >
>> > @wuchong, @fhueske @Fabian
>> >
>> > Best regards,
>> > Jinkui Shi
>> >
>> >
>> > .
>> > ŠĀŠ¤Š¤ LICENSE
>> > ŠĀŠ¤Š¤ NOTICE
>> > ŠĀŠ¤Š¤ README.txt
>> > ŠĀŠ¤Š¤ bin
>> > ŠĻ   ŠĀŠ¤Š¤ config.sh
>> > ŠĻ   ŠĀŠ¤Š¤ flink
>> > ŠĻ   ŠĀŠ¤Š¤ ...
>> > ŠĀŠ¤Š¤ conf
>> > ŠĻ   ŠĀŠ¤Š¤ flink-conf.yaml
>> > ŠĻ   ŠĀŠ¤Š¤ ...
>> > ŠĀŠ¤Š¤ examples
>> > ŠĻ   ŠĀŠ¤Š¤ batch
>> > ŠĻ   Š¸Š¤Š¤ streaming
>> > ŠĀŠ¤Š¤ lib
>> > ŠĻ   ŠĀŠ¤Š¤ flink-dist_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-python_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ ...
>> > ŠĀŠ¤Š¤ log
>> > ŠĀŠ¤Š¤ opt
>> > ŠĻ   ŠĀŠ¤Š¤ flink-cep-scala_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-cep_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-gelly-examples_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-gelly-scala_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-gelly_2.11-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-metrics-dropwizard-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-metrics-ganglia-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-metrics-graphite-1.3.0.jar
>> > ŠĻ   ŠĀŠ¤Š¤ flink-metrics-statsd-1.3.0.jar
>> > ŠĻ   Š¸Š¤Š¤ flink-ml_2.11-1.3.0.jar
>> > ŠĀŠ¤Š¤ resources
>> > ŠĻ   Š¸Š¤Š¤ python
>> > Š¸Š¤Š¤ tools
>> >Š¸Š¤Š¤ planVisualizer.html
>> >
>> >
>> > [1] https://github.com/apache/flink/pull/2460
>> >
>>
>>

[DISCUSS] Flink dist directory management

[shijinkui]

Hi, all

The Flink distribution’s directory have no very clear responsibility about what 
type of files should be in which directory.

The "bin","conf","lib" directories are clear for their responsibility.

But the “opt" directories are mixed with library jars and example jars.

I think we can discuss how is reasonable for the directory. Once we determined, 
we should follow it.

IMO, directory style below is reasonable:

- "examples" directory only contain example jars
- "opt" directory only contain optional library jars in runtime
- "lib" directory only contain library jar that must be loaded at runtime
- “resources” directory only contain resource file used at runtime, such as web 
file

Show your opinion please.

@wuchong, @fhueske @Fabian

Best regards,
Jinkui Shi


.
├── LICENSE
├── NOTICE
├── README.txt
├── bin
│   ├── config.sh
│   ├── flink
│   ├── ...
├── conf
│   ├── flink-conf.yaml
│   ├── ...
├── examples
│   ├── batch
│   └── streaming
├── lib
│   ├── flink-dist_2.11-1.3.0.jar
│   ├── flink-python_2.11-1.3.0.jar
│   ├── ...
├── log
├── opt
│   ├── flink-cep-scala_2.11-1.3.0.jar
│   ├── flink-cep_2.11-1.3.0.jar
│   ├── flink-gelly-examples_2.11-1.3.0.jar
│   ├── flink-gelly-scala_2.11-1.3.0.jar
│   ├── flink-gelly_2.11-1.3.0.jar
│   ├── flink-metrics-dropwizard-1.3.0.jar
│   ├── flink-metrics-ganglia-1.3.0.jar
│   ├── flink-metrics-graphite-1.3.0.jar
│   ├── flink-metrics-statsd-1.3.0.jar
│   └── flink-ml_2.11-1.3.0.jar
├── resources
│   └── python
└── tools
└── planVisualizer.html


[1] https://github.com/apache/flink/pull/2460

Re: Flink as a Service (FaaS)

[shijinkui]

Hi, Chen Qin

We also met your end-to-end use case. A RPC Source and Sink such as netty
source sink can fit such requirements. I’ve submit a natty module in
bahir-flink project which only a demo.
If use connector source instead of Kafka, how do we make the data
persistent? One choice is distributedlog project developed by twitter.

The idea of micro service is very good. Playframework is better choice to
provide micro-service of Flink instead of Flink Monitor which implemented
by netty.
Submit Flink job in the Mesos cluster, at the same time deploy the
micro-service by marathon to the same Mesos cluster, and enable mesos-dns
for service discovery.

The the micro-service can be a API Gateway for:
1. receiving data from device
2. Sending the data to the Flink Job Source(Netty Source with
distributedlog)
3. At same time, the sink send the streaming result data to the API Gateway
4. API Gateway support streaming invoke: send the sink result data to the
device channel

So this plan can guarantee the end-user invoke the service synchronized，
and don’t care about Flink Job’s data processing.

By the way, X as a Service actually is called by SAAS/PAAS in the cloud
platform, such as AWS/Azure. We can call it Flink micro service.:)

Best Regards
Jinkui Shi

在 2017/3/14 下午2:13， "Chen Qin"  写入:

>Hi there,
>
>I am very happy about Flink 1.2 release. It was much more robust and
>feature rich compare to previous versions. In the following section, I
>would like to discuss a non typical use case in flink community.
>
>With ever increasing popularity of micro services[1] to scale out popular
>online services. Various aspect of source of truth is stored (a.k.a
>partitioned) behind various of service rpc endpoints. There is a general
>need of managing events traversal and enrichment throughout org SOA
>systems. (SOA) It is no longer part of data infrastructure scope, where
>traditionally known as batched slow and analytic(small % lossy is okay).
>Flink might also find a fit into core services as well.
>
>It's part of online production services, serving directly from mobile
>client events more importantly services database post commit logs and
>orchestrate adhoc stream toplogies to transform and transfer between
>online
>services(usually backed by databases and serving request response with
>stragent latency requirement)
>
>Use case:
>user updates comes from mobile client via kafka topic, which consumed both
>by user service as well as streaming job. When streaming job do RPC and
>trying to enrich user information, it cause race condition which turns out
>database persistence is not as speedy as streaming job.
>
>In general, streaming job should consume user service commit logs instead
>of karfka topic which defines as source of truth in term of user
>information. Is there a general way to couple with these issues?
>
>P.S I was able to build task manager as jar package and deployed to
>production environment. Instead of using YARN to manage warehouse
>machines.
>Utilize same deployment environment as other online services as docker. So
>far, it seems running smoothly.
>
>Thanks,
>Chen
>
>
>[1] https://en.wikipedia.org/wiki/Microservices
>[2] https://martinfowler.com/eaaDev/EventSourcing.html

[jira] [Created] (FLINK-5860) Replace all the file creating from java.io.tmpdir with TemporaryFolder

[shijinkui (JIRA)]

shijinkui created FLINK-5860:


 Summary: Replace all the file creating from java.io.tmpdir with 
TemporaryFolder
 Key: FLINK-5860
 URL: https://issues.apache.org/jira/browse/FLINK-5860
 Project: Flink
  Issue Type: Test
  Components: Tests
Reporter: shijinkui


Search `System.getProperty("java.io.tmpdir")` whole Flink project. It will get 
a  Unit test list. Replace all the file creating from `java.io.tmpdir` with 
TemporaryFolder.

Who can fix this problem thoroughly?

```

$ grep -ri 'System.getProperty("java.io.tmpdir")' .
./flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/example/CassandraTupleWriteAheadSinkExample.java:
env.setStateBackend(new FsStateBackend("file:///" + 
System.getProperty("java.io.tmpdir") + "/flink/backend"));
./flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java:
  File tempDir = new File(System.getProperty("java.io.tmpdir"));
./flink-connectors/flink-connector-kafka-0.8/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java:
   File tempDir = new File(System.getProperty("java.io.tmpdir"));
./flink-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java:
   File tempDir = new File(System.getProperty("java.io.tmpdir"));
./flink-contrib/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackendConfigTest.java:
   return getMockEnvironment(new File[] { new 
File(System.getProperty("java.io.tmpdir")) });
./flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java: 
public static final String DEFAULT_TASK_MANAGER_TMP_PATH = 
System.getProperty("java.io.tmpdir");
./flink-core/src/test/java/org/apache/flink/api/common/io/EnumerateNestedFilesTest.java:
final String tempPath = System.getProperty("java.io.tmpdir");
./flink-core/src/test/java/org/apache/flink/testutils/TestConfigUtils.java: 
final File tempDir = new File(System.getProperty("java.io.tmpdir"));
./flink-core/src/test/java/org/apache/flink/testutils/TestFileUtils.java:   
File tempDir = new File(System.getProperty("java.io.tmpdir"));
./flink-core/src/test/java/org/apache/flink/testutils/TestFileUtils.java:   
File tempDir = new File(System.getProperty("java.io.tmpdir"));
./flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/util/KMeansDataGenerator.java:
final String outDir = params.get("output", 
System.getProperty("java.io.tmpdir"));
./flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/ml/util/LinearRegressionDataGenerator.java:
  final String tmpDir = System.getProperty("java.io.tmpdir");
./flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/relational/util/WebLogDataGenerator.java:
final String outPath = System.getProperty("java.io.tmpdir");
./flink-java8/src/test/java/org/apache/flink/runtime/util/JarFileCreatorLambdaTest.java:
File out = new File(System.getProperty("java.io.tmpdir"), 
"jarcreatortest.jar");
./flink-java8/src/test/java/org/apache/flink/runtime/util/JarFileCreatorLambdaTest.java:
File out = new File(System.getProperty("java.io.tmpdir"), 
"jarcreatortest.jar");
./flink-java8/src/test/java/org/apache/flink/runtime/util/JarFileCreatorLambdaTest.java:
File out = new File(System.getProperty("java.io.tmpdir"), 
"jarcreatortest.jar");
./flink-java8/src/test/java/org/apache/flink/runtime/util/JarFileCreatorLambdaTest.java:
File out = new File(System.getProperty("java.io.tmpdir"), 
"jarcreatortest.jar");
./flink-libraries/flink-python/src/main/java/org/apache/flink/python/api/PythonPlanBinder.java:
 public static final String FLINK_PYTHON_FILE_PATH = 
System.getProperty("java.io.tmpdir") + File.separator + "flink_plan";
./flink-libraries/flink-python/src/main/java/org/apache/flink/python/api/PythonPlanBinder.java:
 public static final String FLINK_TMP_DATA_DIR = 
System.getProperty("java.io.tmpdir") + File.separator + "flink_data";
./flink-libraries/flink-python/src/main/java/org/apache/flink/python/api/PythonPlanBinder.java:
 FLINK_HDFS_PATH = "file:" + 
System.getProperty("java.io.tmpdir") + File.separator + "flink";
./flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils

[jira] [Created] (FLINK-5839) Flink Security in Huawei's use case

[shijinkui (JIRA)]

shijinkui created FLINK-5839:


 Summary: Flink Security in Huawei's use case
 Key: FLINK-5839
 URL: https://issues.apache.org/jira/browse/FLINK-5839
 Project: Flink
  Issue Type: Improvement
Reporter: shijinkui


This issue collect some security problem found in huawei's use case.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-5806) TaskExecutionState toString format have wrong key

[shijinkui (JIRA)]

shijinkui created FLINK-5806:


 Summary: TaskExecutionState toString format have wrong key
 Key: FLINK-5806
 URL: https://issues.apache.org/jira/browse/FLINK-5806
 Project: Flink
  Issue Type: Bug
Reporter: shijinkui
Assignee: shijinkui


The key of jobID should be executionId in the string format.

public String toString() {
return String.format("TaskState jobId=%s, jobID=%s, state=%s, 
error=%s",
jobID, executionId, executionState,
throwable == null ? "(null)" : 
throwable.toString());
}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-5754) released tag missing .gitigonore .travis.yml .gitattributes

[shijinkui (JIRA)]

shijinkui created FLINK-5754:


 Summary: released tag missing .gitigonore  .travis.yml 
.gitattributes
 Key: FLINK-5754
 URL: https://issues.apache.org/jira/browse/FLINK-5754
 Project: Flink
  Issue Type: Bug
  Components: Build System
Reporter: shijinkui


released tag missing .gitigonore  .travis.yml .gitattributes.
When make a release version, should only replace the version.
for example: https://github.com/apache/spark/tree/v2.1.0




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

答复: [DISCUSS] Development of SQL OVER / Table API Row Windows for streaming tables

[shijinkui]

hi，Fabian, sunjincheng

Today is the first workday of 2017 in China. When we come back, I found the SQL 
issues had been assigned between New Year... 
Yuhong Hong is interest in FLINK-5657. She had implemented it before. Can we 
reconsider to assign FLINK-5657 to her?

Thanks 
Jinkui Shi

[1] https://issues.apache.org/jira/browse/FLINK-4557

-邮件原件-
发件人: Fabian Hueske [mailto:fhue...@gmail.com] 
发送时间: 2017年1月25日 17:55
收件人: dev@flink.apache.org
主题: Re: [DISCUSS] Development of SQL OVER / Table API Row Windows for streaming 
tables

Hi everybody,

thanks for the great discussions so far. It's awesome to see so much interest 
in this topic!

First, I'd like to comment on the development process for this feature and 
later on the design of the runtime:

Dev Process

@Shaoxuan, I completely agree with you. We should first come up with good 
designs for the runtime operators of the different window types. Once we have 
that, we can start implementing the operators and integrate them with Calcite's 
optimization. This will be an intermediate step and as a byproduct give us 
support for SQL OVER windows. Once this is done, we can extend the Table API 
and translate the Table API calls into the same RelNodes as Calcite's SQL 
parser does.

Runtime Design

I think it makes sense to distinguish the different types of OVER windows 
because they have different requirements which result in different runtime 
implementations (with different implementation complexity and performance).
In a previous mail I proposed to split the support for OVER windows into the 
following subtasks:

# bounded PRECEDING
- OVER ROWS for processing time
  - does not require sorted state (data always arrives in processing time
order)
  - no need to consider retraction (processing time is never late)
  - defines windows on row count.
  - A GlobalWindow with evictor + trigger might be the best implementation 
(basically the same as DataStream.countWindow(long, long). We need to add 
timeouts to clean up state for non-used keys though.

- OVER RANGE for processing time
  - does not require sorted state (data always arrives in processing time
order)
  - no need to consider retraction (processing time is never late)
  - defines windows on row count
  - I think this could also be implemented with a GlobalWindow with evictor
+ trigger (need to verify)

- OVER RANGE for event time
  - need for sorted state (late data possible)
  - IMO, a ProcessFunction gives us the most flexibility in adding later 
features (retraction, update rate, etc.)
  - @Shaoxuan, you sketched a good design. Would you like to continue with a 
design proposal?

# UNBOUNDED PRECEDING
Similar considerations apply for the UNBOUNDED PRECEDING cases of the above 
window types.

If we all agree that the separation into six JIRAs (bounded/unbounded * 
row-pt/range-pt/ range-et) makes sense, I would suggest to move the discussions 
about the design of the implementation to the individual JIRAs.

What do think?

Best, Fabian

2017-01-25 9:19 GMT+01:00 Shaoxuan Wang :

> Hi Liuxinchun,
> I am not sure where did you get the inception: anyone has suggested 
> "to process Event time window in Sliding Row Window". If you were 
> referring my post, there may be some misunderstanding there. I think 
> you were asking the similar question as Hongyuhong. I have just 
> replied to him. Please take a look and let me know if that makes sense 
> to you. "Retraction" is an important building block to compute correct 
> incremental results in streaming. It is another big topic, we should 
> discuss this in another thread.
>
> Regards,
> Shaoxuan
>
>
>
> On Wed, Jan 25, 2017 at 3:44 PM, liuxinchun  wrote:
>
> > I don't think it is a good idea to process Event time window in 
> > Sliding Row Window. In Sliding Time window, when an element is late, 
> > we can
> trigger
> > the recalculation of the related windows. And the sliding period is 
> > coarse-gained, We only need to recalculate size/sliding number of
> windows.
> > But in Sliding Row Window, the calculation is triggered when every
> element
> > is coming. The sliding period is becoming fine-gained. When an 
> > element is late, there are so many "windows" are influenced. Even if 
> > we store all
> the
> > raw data, the computation is very large.
> >
> > I think if it is possible to set a standard to sliding Event Time 
> > Row Window, When certain elements are late, we can only recalculate 
> > partial windows and permit some error. For example, we can only 
> > recalculate the windows end in range between (lateElement.timestamp 
> > - leftDelta, lateElement.timestamp] and those windows begin in range 
> > between [lateElement.timestamp, lateElement.timestamp + rightDelta).
> > 
> > //
> >  Hi everyone,
> > Thanks for this great discussion, and glad to see more and more 
> > people
> are
> > interested 

[jira] [Created] (FLINK-5705) webmonitor's request/response use UTF-8 explicitly

[shijinkui (JIRA)]

shijinkui created FLINK-5705:


 Summary: webmonitor's request/response use UTF-8 explicitly
 Key: FLINK-5705
 URL: https://issues.apache.org/jira/browse/FLINK-5705
 Project: Flink
  Issue Type: Improvement
  Components: Webfrontend
Reporter: shijinkui


QueryStringDecoder and HttpPostRequestDecoder use UTF-8 defined in flink.

Response set content-encoding header with utf-8



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-5650) flink-python unit test costs more than half hour

[shijinkui (JIRA)]

shijinkui created FLINK-5650:


 Summary: flink-python unit test costs more than half hour
 Key: FLINK-5650
 URL: https://issues.apache.org/jira/browse/FLINK-5650
 Project: Flink
  Issue Type: Bug
  Components: Python API
Reporter: shijinkui


When execute `mvn clean test` in flink-python, it will wait more than half hour 
after the console output below:
---
 T E S T S
---
Running org.apache.flink.python.api.PythonPlanBinderTest
log4j:WARN No appenders could be found for logger 
(org.apache.flink.python.api.PythonPlanBinderTest).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.



The stack below:
"main" prio=5 tid=0x7f8d7780b800 nid=0x1c03 waiting on condition 
[0x79fd8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at 
org.apache.flink.python.api.streaming.plan.PythonPlanStreamer.startPython(PythonPlanStreamer.java:70)
at 
org.apache.flink.python.api.streaming.plan.PythonPlanStreamer.open(PythonPlanStreamer.java:50)
at 
org.apache.flink.python.api.PythonPlanBinder.startPython(PythonPlanBinder.java:211)
at 
org.apache.flink.python.api.PythonPlanBinder.runPlan(PythonPlanBinder.java:141)
at 
org.apache.flink.python.api.PythonPlanBinder.main(PythonPlanBinder.java:114)
at 
org.apache.flink.python.api.PythonPlanBinderTest.testProgram(PythonPlanBinderTest.java:83)
at 
org.apache.flink.test.util.JavaProgramTestBase.testJobWithoutObjectReuse(JavaProgramTestBase.java:174)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5640) configure the explicit Unit Test file suffix

[shijinkui (JIRA)]

shijinkui created FLINK-5640:


 Summary: configure the explicit Unit Test file suffix
 Key: FLINK-5640
 URL: https://issues.apache.org/jira/browse/FLINK-5640
 Project: Flink
  Issue Type: Test
  Components: Tests
Reporter: shijinkui
Assignee: shijinkui



There are four types of Unit Test file: *ITCase.java, *Test.java, 
*ITSuite.scala, *Suite.scala
File name ending with "IT.java" is integration test. File name ending with 
"Test.java"  is unit test.

It's clear for Surefire plugin of default-test execution to declare that 
"*Test.*" is Java Unit Test.

* Suite  total: 10
* ITCase  total: 378
* Test  total: 1008
* ITSuite  total: 14



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5543) customCommandLine tips in CliFrontend

[shijinkui (JIRA)]

shijinkui created FLINK-5543:


 Summary: customCommandLine tips in CliFrontend
 Key: FLINK-5543
 URL: https://issues.apache.org/jira/browse/FLINK-5543
 Project: Flink
  Issue Type: Improvement
  Components: Client
Reporter: shijinkui


Tips: DefaultCLI must be added at the end, because 
getActiveCustomCommandLine(..) will get the active CustomCommandLine in order 
and DefaultCLI isActive always return true.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5519) scala-maven-plugin version all change to 3.2.2

[shijinkui (JIRA)]

shijinkui created FLINK-5519:


 Summary: scala-maven-plugin version all change to 3.2.2
 Key: FLINK-5519
 URL: https://issues.apache.org/jira/browse/FLINK-5519
 Project: Flink
  Issue Type: Improvement
  Components: Build System
Reporter: shijinkui


1. scala-maven-plugin version all change to 3.2.2 in all module
2. parent pom version change to apache-18 from apache-14



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

答复: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release candidate)

[shijinkui]

Do we have a probable time of 1.2 release? This month or Next month?

-邮件原件-
发件人: Robert Metzger [mailto:rmetz...@apache.org] 
发送时间: 2017年1月3日 20:44
收件人: dev@flink.apache.org
抄送: u...@flink.apache.org
主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release candidate)

Hi,

First of all, I wish everybody a happy new year 2017.

I've set user@flink in CC so that users who are interested in helping with the 
testing get notified. Please respond only to the dev@ list to keep the 
discussion there!

According to the 1.2 release discussion thread, I've created a first release 
candidate for Flink 1.2.
The release candidate will not be the final release, because I'm certain that 
we'll find at least one blocking issue in the candidate :)

Therefore, the RC is meant as a testing only release candidate.
Please report every issue we need to fix before the next RC in this thread so 
that we have a good overview.

The release artifacts are located here:
http://people.apache.org/~rmetzger/flink-1.2.0-rc0/

The maven staging repository is located here:
https://repository.apache.org/content/repositories/orgapacheflink-

The release commit (in branch "release-1.2.0-rc0"):
http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced


Happy testing!

[jira] [Created] (FLINK-5399) Add more information to checkpoint result of TriggerSavepointSuccess

[shijinkui (JIRA)]

shijinkui created FLINK-5399:


 Summary: Add more information to checkpoint result of 
TriggerSavepointSuccess
 Key: FLINK-5399
 URL: https://issues.apache.org/jira/browse/FLINK-5399
 Project: Flink
  Issue Type: Improvement
  Components: State Backends, Checkpointing
Reporter: shijinkui


Add checkpointId and triggerTime to TriggerSavepointSuccess

We can record the history of trigger checkpoint out of Flink System.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5396) flink-dist replace scala version in opt.xml by change-scala-version.sh

[shijinkui (JIRA)]

shijinkui created FLINK-5396:


 Summary: flink-dist replace scala version in opt.xml by 
change-scala-version.sh
 Key: FLINK-5396
 URL: https://issues.apache.org/jira/browse/FLINK-5396
 Project: Flink
  Issue Type: Improvement
Reporter: shijinkui


flink-dist have configured for replacing bin.xml, but not opt.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5395) support locally build distribution by script create_release_files.sh

[shijinkui (JIRA)]

shijinkui created FLINK-5395:


 Summary: support locally build distribution by script 
create_release_files.sh
 Key: FLINK-5395
 URL: https://issues.apache.org/jira/browse/FLINK-5395
 Project: Flink
  Issue Type: Improvement
  Components: Build System
Reporter: shijinkui


create_release_files.sh is build flink release only. It's hard to build custom 
local Flink release distribution.

Let create_release_files.sh support:
1. custom git repo url
2. custom build special scala and hadoop version
3. fix flink-dist opt.xml have no replace the scala version by 
change-scala-version.sh



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5370) build failure for unit test of FileInputFormatTest and GlobFilePathFilterTest

[shijinkui (JIRA)]

shijinkui created FLINK-5370:


 Summary: build failure for unit test of FileInputFormatTest and 
GlobFilePathFilterTest
 Key: FLINK-5370
 URL: https://issues.apache.org/jira/browse/FLINK-5370
 Project: Flink
  Issue Type: Test
  Components: Tests
Reporter: shijinkui


mvn clean package

head commit 4a27d2105dd08f323c0be26e79a55986aa97e7bd


Results :

Failed tests:
  FileInputFormatTest.testExcludeFiles:336 Illegal char <:> at index 2: 
/C:/Users/S00383~1/AppData/Local/Temp/junit2200257114857246164/anot
her_file.bin
  FileInputFormatTest.testReadMultiplePatterns:369 Illegal char <:> at index 2: 
/C:/Users/S00383~1/AppData/Local/Temp/junit1476821885889426
068/another_file.bin
Tests in error:
  GlobFilePathFilterTest.excludeFilenameWithStart:115 ? InvalidPath Illegal 
char...

Tests run: 2084, Failures: 2, Errors: 1, Skipped: 0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5217) Deprecated interface Checkpointed make clear suggestion

[shijinkui (JIRA)]

shijinkui created FLINK-5217:


 Summary: Deprecated interface Checkpointed make clear suggestion
 Key: FLINK-5217
 URL: https://issues.apache.org/jira/browse/FLINK-5217
 Project: Flink
  Issue Type: Improvement
  Components: DataStream API
Reporter: shijinkui



package org.apache.flink.streaming.api.checkpoint;
@Deprecated
@PublicEvolving
public interface Checkpointed extends 
CheckpointedRestoring

this interface should have clear suggestion which version to give up this 
interface, and which interface can instead of it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5168) Scaladoc annotation link use [[]] instead of {@link}

[shijinkui (JIRA)]

shijinkui created FLINK-5168:


 Summary: Scaladoc annotation link use [[]] instead of {@link}
 Key: FLINK-5168
 URL: https://issues.apache.org/jira/browse/FLINK-5168
 Project: Flink
  Issue Type: Improvement
  Components: Scala API
Reporter: shijinkui


{@link StreamExecutionEnvironment#readFile(FileInputFormat,
  *  String, FileProcessingMode, long)}

==>

[[StreamExecutionEnvironment#readFile(FileInputFormat, String, 
FileProcessingMode, long)]]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5167) StreamExecutionEnvironment's set function return `this` instead of void

[shijinkui (JIRA)]

shijinkui created FLINK-5167:


 Summary: StreamExecutionEnvironment's set function return `this` 
instead of void
 Key: FLINK-5167
 URL: https://issues.apache.org/jira/browse/FLINK-5167
 Project: Flink
  Issue Type: Bug
  Components: DataStream API
Reporter: shijinkui


for example :

public void setNumberOfExecutionRetries(int numberOfExecutionRetries) {
config.setNumberOfExecutionRetries(numberOfExecutionRetries);
}

change to:

public StreamExecutionEnvironment setNumberOfExecutionRetries(int 
numberOfExecutionRetries) {
config.setNumberOfExecutionRetries(numberOfExecutionRetries);
return this;
}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5166) TextInputFormatTest.testNestedFileRead

[shijinkui (JIRA)]

shijinkui created FLINK-5166:


 Summary: TextInputFormatTest.testNestedFileRead
 Key: FLINK-5166
 URL: https://issues.apache.org/jira/browse/FLINK-5166
 Project: Flink
  Issue Type: Bug
  Components: Build System
Reporter: shijinkui


mvn clean package -P \!scala-2.11,scala-2.11  -U

Failed tests:
  TextInputFormatTest.testNestedFileRead:140 Test erroneous

Tests run: 846, Failures: 1, Errors: 0, Skipped: 0





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[Discuss] State Backend use external HBase storage

[shijinkui]

Hi, All

At present flink have three state backend: memory, file system, rocksdb.
MemoryStateBackend will tansform the snapshot to jobManager, 5MB limited 
default. Even setting it bigger, that not suitable for very big state storage.
HDFS can meet the reliability guarantee, but It's slow. File System and RocksDB 
are fast, but they are have no reliability guarantee.
Three state backend all have no reliability guarantee.

Can we have a Hbase state backend, providing reliability guarantee of state 
snapshot?
For user, only new a HbaseStateBackend object, provide hbase parameter and 
optimization configure.
Maybe Hbase or other distributed key-value storage is heavyweight storage, we 
only use hbase client to read/write asynchronously.

-Jinkui Shi

scala api createLocalEnvironment() function add default Configuration parameter

[shijinkui]

Hi，all

scala program can't direct use createLocalEnvironment with custom Configure 
object.

such as I want to start web server in local mode with Flink UI, I will do such 
as:

```
// set up execution environment
val conf = new Configuration
conf.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true)
conf.setInteger(ConfigConstants.JOB_MANAGER_WEB_PORT_KEY, 
ConfigConstants.DEFAULT_JOB_MANAGER_WEB_FRONTEND_PORT)
val env = new org.apache.flink.streaming.api.scala.StreamExecutionEnvironment(
  
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.createLocalEnvironment(2,
 conf)
)
```

For using conveniently and perfect API, we need createLocalEnvironment function 
have a config parameter

detail in FLINK-4669.

-Jinkui Shi

答复: 答复: [DISCUSS] add netty tcp/restful pushed source support

[shijinkui]

It's nice. Will present flink source connector be pushed to bahir-flink?
I can add netty-source to bahir-flink.

Maven repository have no bahir-flink's.
https://mvnrepository.com/artifact/org.apache.bahir

-邮件原件-
发件人: Greg Hogan [mailto:c...@greghogan.com] 
发送时间: 2016年9月27日 20:58
收件人: dev@flink.apache.org
主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support

Apache Bahir's website only suggests support for additional frameworks, but 
there is a Flink repository at
  https://github.com/apache/bahir-flink

On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <shijin...@huawei.com> wrote:

> Hey, Stephan Ewen
>
> 1.  bahir's target is spark. The contributer are rxin, srowen, tdas,
> mateiz and so on.
> If we want bahir used by flink, we can suggest bahir provide 
> streaming connecter interface, such as store(), start(), stop(), 
> restart(), receiving(Any)...
> Then same streaming connector can be implemented by spark and 
> flink. But I think this is impossible, as bahir depend spark-streaming 
> and spark sql.
> 2.  About connector fault tolerance. Bahir's mqtt and akka connector
> are themselves' storage. But netty have no persist data feature.
> I think we can append data to a ringbuffer. When SourceContext
> collect() throw error, then write message to inform client to stop 
> send message. When flink SourceContext is normal, then write the 
> ringbuffer data to flink, inform client to go on.
> Because pushing mode is hard to control the flow throughput, 
> the upstream client can.
> This netty connector's purpose is end-to-end streaming, 
> minimum time delay.
> 3.  Later on, We can provide http protocol ability, put, post, get,
> even websocket or jersey restful based on netty.
>
> -邮件原件-
> 发件人: Stephan Ewen [mailto:se...@apache.org]
> 发送时间: 2016年9月27日 16:54
> 收件人: dev@flink.apache.org
> 主题: Re: [DISCUSS] add netty tcp/restful pushed source support
>
> I think that could be an interesting source. Two quick questions to 
> move forward
>
>   - To keep the Flink code base from becoming too big (hard to 
> maintain and
> test) we started working with Apache Bahir as a project dedicated to 
> streaming connectors. Would that be a good target for the connector?
>
>   - What are your thoughts on fault tolerance for that connector?
>
> On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <shijin...@huawei.com> wrote:
>
> > Hi, all
> >
> > 1.In order to support end-to-end pushed source, I create FLINK-4630<
> > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know 
> > whether is this idea worth?
> >
> > ---
> > When source stream get start, listen a provided tcp port, receive 
> > stream data from user data source.
> > This netty tcp source is keepping alive and end-to-end, that is from 
> > business system to flink worker directly.
> >
> > user app push ->  netty server source of Flink
> >
> > describe the source in detail below:
> >
> > 1.source run as a netty tcp server
> > 2.user provide a tcp port, if the port is in used, increace the port
> > number between 1024 to 65535. Source can parallel.
> > 3.callback the provided url to report the real port to listen
> > 4.user push streaming data to netty server, then collect the data to
> > flink
> >
> >
> > Thanks
> >
> > Jinkui Shi
> >
> >
>

答复: [DISCUSS] add netty tcp/restful pushed source support

[shijinkui]

Hey, Stephan Ewen

1.  bahir's target is spark. The contributer are rxin, srowen, tdas, mateiz 
and so on.
If we want bahir used by flink, we can suggest bahir provide streaming 
connecter interface, such as store(), start(), stop(), restart(), 
receiving(Any)...
Then same streaming connector can be implemented by spark and flink. 
But I think this is impossible, as bahir depend spark-streaming and spark sql.
2.  About connector fault tolerance. Bahir's mqtt and akka connector are 
themselves' storage. But netty have no persist data feature. 
I think we can append data to a ringbuffer. When SourceContext 
collect() throw error, then write message to inform client to stop send 
message. When flink SourceContext is normal, then write the ringbuffer data to 
flink, inform client to go on.
Because pushing mode is hard to control the flow throughput, the 
upstream client can.
This netty connector's purpose is end-to-end streaming, minimum time 
delay.
3.  Later on, We can provide http protocol ability, put, post, get, even 
websocket or jersey restful based on netty.

-邮件原件-
发件人: Stephan Ewen [mailto:se...@apache.org] 
发送时间: 2016年9月27日 16:54
收件人: dev@flink.apache.org
主题: Re: [DISCUSS] add netty tcp/restful pushed source support

I think that could be an interesting source. Two quick questions to move forward

  - To keep the Flink code base from becoming too big (hard to maintain and
test) we started working with Apache Bahir as a project dedicated to streaming 
connectors. Would that be a good target for the connector?

  - What are your thoughts on fault tolerance for that connector?

On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <shijin...@huawei.com> wrote:

> Hi, all
>
> 1.In order to support end-to-end pushed source, I create FLINK-4630<
> https://issues.apache.org/jira/browse/FLINK-4630>. I want to know 
> whether is this idea worth?
>
> ---
> When source stream get start, listen a provided tcp port, receive 
> stream data from user data source.
> This netty tcp source is keepping alive and end-to-end, that is from 
> business system to flink worker directly.
>
> user app push ->  netty server source of Flink
>
> describe the source in detail below:
>
> 1.source run as a netty tcp server
> 2.user provide a tcp port, if the port is in used, increace the port
> number between 1024 to 65535. Source can parallel.
> 3.callback the provided url to report the real port to listen
> 4.user push streaming data to netty server, then collect the data to
> flink
>
>
> Thanks
>
> Jinkui Shi
>
>

[DISCUSS] add netty tcp/restful pushed source support

[shijinkui]

Hi, all

1.In order to support end-to-end pushed source, I create 
FLINK-4630. I want to know 
whether is this idea worth?

---
When source stream get start, listen a provided tcp port, receive stream data 
from user data source.
This netty tcp source is keepping alive and end-to-end, that is from business 
system to flink worker directly.

user app push ->  netty server source of Flink

describe the source in detail below:

1.source run as a netty tcp server
2.user provide a tcp port, if the port is in used, increace the port number 
between 1024 to 65535. Source can parallel.
3.callback the provided url to report the real port to listen
4.user push streaming data to netty server, then collect the data to flink


Thanks

Jinkui Shi

答复: 答复: [discuss] merge module flink-yarn and flink-yarn-test

[shijinkui]

Hi，Maximilian Michels

Thank for your reply.

In order to test submit Flink job to yarn cluster, that is testing client and 
session cli, split unit test from flink-yarn.
First of all, such yarn cluster unit test should be rename 'flink-yarn-tests' 
as 'flink-yarn-cluster-test', and move to 'flink-tests' folder as its 
sub-module.

I think unit test can be divide into module's unit test which can be execute 
when module is building and unit test module which should be executed 
independently.
The top module in the root fold shouldn't be a unit test module, not only for 
good-looking, for its duty.

-邮件原件-
发件人: Maximilian Michels [mailto:m...@apache.org] 
发送时间: 2016年9月26日 16:33
收件人: dev@flink.apache.org
主题: Re: 答复: [discuss] merge module flink-yarn and flink-yarn-test

Hello Jinkui Shi,

Due to the nature of most of the Yarn tests, we need them to be in a separate 
module. More concretely, these tests have a dependency on 'flink-dist' because 
they need to deploy the Flink fat jar to the Yarn tests cluster. The fat jar 
also contains the 'flink-yarn' code. Thus, 'flink-yarn' needs to be a separate 
module and built before 'flink-yarn-tests'.

That being said, some of the tests don't need the fat jar, so we could move 
some of the tests to 'flink-yarn'. However, that is mostly a cosmetic change 
and not important for the testing coverage.

Best,
Max

On Thu, Sep 22, 2016 at 12:26 PM, Stephan Ewen <se...@apache.org> wrote:
>
> "flink-test-utils" contains, as the name says, utils for testing. 
> Intended to be used by users in writing their own tests.
> "flink-tests" contains cross module tests, no user should ever need to 
> have a dependency on that.
>
> They are different because users explicitly asked for test utils to be 
> factored into a separate project.
>
> As an honest reply here: Setting up a project as huge as Flink need to 
> take many things into account
>
>   - Multiple languages (Java / Scala), with limitations of IDEs in mind
>   - Dependency conflicts and much shading magic
>   - Dependency matrices (multiple hadoop and scala versions)
>   - Supporting earlier Java versions
>   - clean scope differentiation, so users can reuse utils and testing 
> code
>
>
> That simply requires some extra modules once in a while. Flink people 
> have worked hard on coming up with a structure that serves the need of 
> the production users and automated build/testing systems. These 
> production user requests are most important to us, and sometimes, we 
> need to take cuts in "beauty of directory structure" to help them.
>
> Constantly accusing the community of creating bad structures before 
> even trying to understand the reasoning behind that does not come 
> across as very friendly. Constantly accusing the community of sloppy 
> work just because your laptop settings are incompatible with the 
> default configuration likewise.
>
> I hope you understand that.
>
>
> On Thu, Sep 22, 2016 at 2:58 AM, shijinkui <shijin...@huawei.com> wrote:
>
> > Hi, Stephan
> >
> > Thanks for your reply.
> >
> > In my mind, Maven-shade-plugin and sbt-assembly both default exclude 
> > test code for the fat jar.
> >
> > In fact, unit tests are use to test the main code, ensure our code 
> > logic fit our expect . This is general convention. I think. Flink 
> > has be a top apache project. We shouldn't be special. We're 
> > programmer, should be professional.
> >
> > Even more, there are `flink-tes-utils-parent` and `flink-tests` 
> > module, what's the relation between them.
> >
> > I have to ask why they are exist? Where is the start of such 
> > confusion modules?
> >
> > I think we shouldn't do nothing for this. Code and design should be 
> > comfortable.
> >
> > Thanks
> >
> > From Jinkui Shi
> >
> > -邮件原件-
> > 发件人: Stephan Ewen [mailto:se...@apache.org]
> > 发送时间: 2016年9月21日 22:19
> > 收件人: dev@flink.apache.org
> > 主题: Re: [discuss] merge module flink-yarn and flink-yarn-test
> >
> > I would like Robert to comment on this.
> >
> > I think there was a reason to have different modules, which had 
> > again something to do with the Maven Shade Plugin Dependencies and 
> > shading really seem the trickiest thing in bigger Java/Scala 
> > projects ;-)
> >
> > On Wed, Sep 21, 2016 at 11:04 AM, shijinkui <shijin...@huawei.com> wrote:
> >
> > > Hi, All
> > >
> > > There too much module in the root. There are no necessary to 
> > > separate the test code from sub-module.
> > >
> > > I never see such design: two modules, one is main code, the other 
> > > is test code.
> > >
> > > Is there some special reason?
> > >
> > > From Jinkui Shi
> > >
> >

[jira] [Created] (FLINK-4669) scala api createLocalEnvironment() function add default Configuration parameter

[shijinkui (JIRA)]

shijinkui created FLINK-4669:


 Summary: scala api createLocalEnvironment() function add default 
Configuration parameter
 Key: FLINK-4669
 URL: https://issues.apache.org/jira/browse/FLINK-4669
 Project: Flink
  Issue Type: Improvement
  Components: Streaming
Reporter: shijinkui


scala program can't direct use createLocalEnvironment with custom Configure 
object.

such as I want to start web server in local mode, I will do such as:

```
// set up execution environment
val conf = new Configuration
conf.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true)
conf.setInteger(ConfigConstants.JOB_MANAGER_WEB_PORT_KEY, 
ConfigConstants.DEFAULT_JOB_MANAGER_WEB_FRONTEND_PORT)
val env = new org.apache.flink.streaming.api.scala.StreamExecutionEnvironment(
  
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.createLocalEnvironment(2,
 conf)
)
```

so we need createLocalEnvironment function have a config parameter 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4661) Failure to find org.apache.flink:flink-runtime_2.10:jar:tests

[shijinkui (JIRA)]

shijinkui created FLINK-4661:


 Summary: Failure to find 
org.apache.flink:flink-runtime_2.10:jar:tests
 Key: FLINK-4661
 URL: https://issues.apache.org/jira/browse/FLINK-4661
 Project: Flink
  Issue Type: Bug
Reporter: shijinkui


[ERROR] Failed to execute goal on project flink-streaming-java_2.10: Could not 
resolve dependencies for project 
org.apache.flink:flink-streaming-java_2.10:jar:1.2-SNAPSHOT: Failure to find 
org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT in 
http://localhost:/repository/maven-public/ was cached in the local 
repository, resolution will not be reattempted until the update interval of 
nexus-releases has elapsed or updates are forced -> [Help 1]


Failure to find org.apache.flink:flink-runtime_2.10:jar:tests

I can't find where this tests jar is generated.

By the way, recently half month, I start to use flink. There is zero time I can 
compile the Flink project with default setting..




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

答复: [discuss] merge module flink-yarn and flink-yarn-test

[shijinkui]

Hi, Stephan

Thanks for your reply. 

In my mind, Maven-shade-plugin and sbt-assembly both default exclude test code 
for the fat jar.

In fact, unit tests are use to test the main code, ensure our code logic fit 
our expect . This is general convention. I think. Flink has be a top apache 
project. We shouldn't be special. We're programmer, should be professional.

Even more, there are `flink-tes-utils-parent` and `flink-tests` module, what's 
the relation between them. 

I have to ask why they are exist? Where is the start of such confusion modules? 

I think we shouldn't do nothing for this. Code and design should be comfortable.

Thanks

From Jinkui Shi

-邮件原件-
发件人: Stephan Ewen [mailto:se...@apache.org] 
发送时间: 2016年9月21日 22:19
收件人: dev@flink.apache.org
主题: Re: [discuss] merge module flink-yarn and flink-yarn-test

I would like Robert to comment on this.

I think there was a reason to have different modules, which had again something 
to do with the Maven Shade Plugin Dependencies and shading really seem the 
trickiest thing in bigger Java/Scala projects ;-)

On Wed, Sep 21, 2016 at 11:04 AM, shijinkui <shijin...@huawei.com> wrote:

> Hi, All
>
> There too much module in the root. There are no necessary to separate 
> the test code from sub-module.
>
> I never see such design: two modules, one is main code, the other is 
> test code.
>
> Is there some special reason?
>
> From Jinkui Shi
>

[discuss] merge module flink-yarn and flink-yarn-test

[shijinkui]

Hi, All

There too much module in the root. There are no necessary to separate the test 
code from sub-module.

I never see such design: two modules, one is main code, the other is test code.

Is there some special reason?

>From Jinkui Shi

[jira] [Created] (FLINK-4630) add netty tcp source support

[shijinkui (JIRA)]

shijinkui created FLINK-4630:


 Summary: add netty tcp source support
 Key: FLINK-4630
 URL: https://issues.apache.org/jira/browse/FLINK-4630
 Project: Flink
  Issue Type: New Feature
  Components: Streaming Connectors
Reporter: shijinkui


When source stream get start, listen a provided tcp port, receive stream data 
sent from the place where origin data generated.
This netty tcp source is end-to-end, that is from business system to flink 
worker directly. 

Such source service is needed in produce indeed.

describe the source in detail below:

1.  source run as a netty tcp server
2.  user provide a tcp port, if the port is in used, increace the port 
number between 1024 to 65535
3.  callback the provided url to report the real port to listen



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4562) table example make an divided module in flink-examples

[shijinkui (JIRA)]

shijinkui created FLINK-4562:


 Summary: table example make an divided module in flink-examples
 Key: FLINK-4562
 URL: https://issues.apache.org/jira/browse/FLINK-4562
 Project: Flink
  Issue Type: Improvement
Reporter: shijinkui


example code should't packaged in table module.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4561) replace all the scala version as a property

[shijinkui (JIRA)]

shijinkui created FLINK-4561:


 Summary: replace all the scala version as a property
 Key: FLINK-4561
 URL: https://issues.apache.org/jira/browse/FLINK-4561
 Project: Flink
  Issue Type: Improvement
Reporter: shijinkui


replace all the scala version(2.10) as a property `scala.binary.version` 
defined in root pom properties. default scala version property is 2.10.

modify:
1. dependency include scala version 
2. module defining include scala version
3. scala version upgrade to 2.11.8 from 2.11.7



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4560) enforcer java version as 1.7

[shijinkui (JIRA)]

shijinkui created FLINK-4560:


 Summary: enforcer java version as 1.7
 Key: FLINK-4560
 URL: https://issues.apache.org/jira/browse/FLINK-4560
 Project: Flink
  Issue Type: Improvement
Reporter: shijinkui


1. maven-enforcer-plugin add java version enforce
2. maven-enforcer-plugin version upgrade to 1.4.1

explicit require java version



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4519) scala maxLineLength increased to 120

[shijinkui (JIRA)]

shijinkui created FLINK-4519:


 Summary: scala maxLineLength increased to 120  
 Key: FLINK-4519
 URL: https://issues.apache.org/jira/browse/FLINK-4519
 Project: Flink
  Issue Type: Improvement
Reporter: shijinkui


`tools/maven/scalastyle-config.xml`



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4518) maxLineLength change to 120

[shijinkui (JIRA)]

shijinkui created FLINK-4518:


 Summary: maxLineLength change to 120
 Key: FLINK-4518
 URL: https://issues.apache.org/jira/browse/FLINK-4518
 Project: Flink
  Issue Type: Improvement
Reporter: shijinkui


scala checkstyle maxLineLength increase to 120 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4517) scala code refactoring

[shijinkui (JIRA)]

shijinkui created FLINK-4517:


 Summary: scala code refactoring
 Key: FLINK-4517
 URL: https://issues.apache.org/jira/browse/FLINK-4517
 Project: Flink
  Issue Type: Improvement
Reporter: shijinkui
Priority: Minor


such as:
1. case class does't need `new`
2. case block does't need `{  ...  }`
3. match-case instead of isInstanceOf 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)