[jira] [Commented] (FLINK-16850) Make flink-benchmark align with different releases of Flink
[
https://issues.apache.org/jira/browse/FLINK-16850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104099#comment-17104099
]
Yun Tang commented on FLINK-16850:
--
[~tzulitai], would you please kindly help to create the 1st commit for the
empty repo: https://github.com/apache/flink-benchmarks to migrate from
https://github.com/dataArtisans/flink-benchmarks ? Thank you very much.
> Make flink-benchmark align with different releases of Flink
> ---
>
> Key: FLINK-16850
> URL: https://issues.apache.org/jira/browse/FLINK-16850
> Project: Flink
> Issue Type: Sub-task
> Components: Benchmarks
>Affects Versions: 1.10.0
>Reporter: Yun Tang
>Priority: Major
>
> Currently, [flink-benchmark|https://github.com/dataArtisans/flink-benchmarks]
> does not have any releases to align with release branches of Flink. Just
> change the {{flink.version}} when executing benchmark is not enough as
> {{flink-benchmark}} always use the latset interfaces which might not existed
> in previous Flink versions. Take FLINK-15222 for example, this let
> {{flink-benchmark}} to use new version of interface which is only existed
> after Flink-1.11
> I think we could also let {{flink-benchmark}} to have releases or at least
> specific branch to align with releases of Flink.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-web] klion26 commented on pull request #267: [FLINK-13682][docs-zh] Translate "Code Style - Scala Guide" page into Chinese
klion26 commented on pull request #267: URL: https://github.com/apache/flink-web/pull/267#issuecomment-626486117 @yangjf2019 thanks for addressing my comments, LGTM ping @wuchong This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] tzulitai opened a new pull request #107: [FLINK-17605] Consolidate flink.version property in parent POM
tzulitai opened a new pull request #107: URL: https://github.com/apache/flink-statefun/pull/107 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17605) Consolidate flink.version in Stateful Functions POM files
[
https://issues.apache.org/jira/browse/FLINK-17605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-17605:
---
Labels: pull-request-available (was: )
> Consolidate flink.version in Stateful Functions POM files
> -
>
> Key: FLINK-17605
> URL: https://issues.apache.org/jira/browse/FLINK-17605
> Project: Flink
> Issue Type: Improvement
> Components: Stateful Functions
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Minor
> Labels: pull-request-available
>
> Right now, we have definitions of {{flink.version}} property scattered in
> multiple POM files in Stateful Functions. They should be consolidated in the
> root parent POM, as that should not likely change across modules.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on pull request #11687: [FLINK-16536][network][checkpointing] Implement InputChannel state recovery for unaligned checkpoint
zhijiangW commented on pull request #11687: URL: https://github.com/apache/flink/pull/11687#issuecomment-626483029 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-17605) Consolidate flink.version in Stateful Functions POM files
Tzu-Li (Gordon) Tai created FLINK-17605:
---
Summary: Consolidate flink.version in Stateful Functions POM files
Key: FLINK-17605
URL: https://issues.apache.org/jira/browse/FLINK-17605
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Right now, we have definitions of {{flink.version}} property scattered in
multiple POM files in Stateful Functions. They should be consolidated in the
root parent POM, as that should not likely change across modules.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zjffdu commented on pull request #11895: [FLINK-10911][scala-shell] Enable flink-scala-shell with Scala 2.12
zjffdu commented on pull request #11895: URL: https://github.com/apache/flink/pull/11895#issuecomment-626482480 @aljoscha I found that the case class issue is caused by `FLINK-10493` which introduce `ScalaCaseClassSerializer`. In our internal flink branch which is based on 1.5 that has no `ScalaCaseClassSerializer` it works even when I use `repl-class-based` in scala shell. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11297: [FLINK-16364] [table] Deprecate the methods in TableEnvironment proposed by FLIP-84
flinkbot edited a comment on pull request #11297: URL: https://github.com/apache/flink/pull/11297#issuecomment-593931358 ## CI report: * bd7ca0b4407e126739f17292b8a6a0561508351e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=935) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12066: [hotfix][runtime] Remove useless local variable in CompletedCheckpointStoreTest3testAddCheckpointMoreThanMaxRetained
flinkbot edited a comment on pull request #12066: URL: https://github.com/apache/flink/pull/12066#issuecomment-626471650 ## CI report: * fa88acd466c5c1d22389d73610009b0c25c69fbb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=939) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12063: [FLINK-17603][table][core] Prepare Hive partitioned streaming source
flinkbot edited a comment on pull request #12063: URL: https://github.com/apache/flink/pull/12063#issuecomment-626441217 ## CI report: * 11d3939cde0deeac80e4642bb946622e7ee4d850 UNKNOWN * 64ffbab011482e857ba986b2116d9dfb94138259 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=928) * b809090fe19a17f34bbf25d20e4ad9bb5a618028 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=938) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12061: [FLINK-17454][python]python gateway callback server port should not b…
flinkbot edited a comment on pull request #12061: URL: https://github.com/apache/flink/pull/12061#issuecomment-626432764 ## CI report: * da1bf1e181793e94b1c9147546e15101e87e066f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=936) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12066: [hotfix][runtime] Remove useless local variable in CompletedCheckpointStoreTest3testAddCheckpointMoreThanMaxRetained
flinkbot commented on pull request #12066: URL: https://github.com/apache/flink/pull/12066#issuecomment-626471650 ## CI report: * fa88acd466c5c1d22389d73610009b0c25c69fbb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12064: [hotfix][javadocs]java doc error fix
flinkbot edited a comment on pull request #12064: URL: https://github.com/apache/flink/pull/12064#issuecomment-626450834 ## CI report: * a799713860db44d58407123f19236b68e231094b Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/165085794) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=930) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
flinkbot edited a comment on pull request #12004: URL: https://github.com/apache/flink/pull/12004#issuecomment-624445192 ## CI report: * 44df52a0ff62e1b4ec65efd6a14a8b07d7712e75 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=872) * 3ac79a41395792dfecefc87a19c58468e91ede8d UNKNOWN * b3802f894dcdd7226316583e66b37a662c3e5bdb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=929) * 6545efc0050c0d2ae84755199cf3b5fd55e82dca Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=937) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12066: [hotfix][runtime] Remove useless local variable in CompletedCheckpointStoreTest3testAddCheckpointMoreThanMaxRetained
flinkbot commented on pull request #12066: URL: https://github.com/apache/flink/pull/12066#issuecomment-626467290 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit fa88acd466c5c1d22389d73610009b0c25c69fbb (Mon May 11 04:49:23 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhengcanbin opened a new pull request #12066: [hotfix][runtime] Remove useless local variable in CompletedCheckpointStoreTest3testAddCheckpointMoreThanMaxRetained
zhengcanbin opened a new pull request #12066: URL: https://github.com/apache/flink/pull/12066 Remove useless local variable `taskStates` in `CompletedCheckpointStoreTest3testAddCheckpointMoreThanMaxRetained` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12061: [FLINK-17454][python]python gateway callback server port should not b…
flinkbot edited a comment on pull request #12061: URL: https://github.com/apache/flink/pull/12061#issuecomment-626432764 ## CI report: * d07c04b5ac978764112af3bf1d35285195e3440d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=924) * da1bf1e181793e94b1c9147546e15101e87e066f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=936) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11297: [FLINK-16364] [table] Deprecate the methods in TableEnvironment proposed by FLIP-84
flinkbot edited a comment on pull request #11297: URL: https://github.com/apache/flink/pull/11297#issuecomment-593931358 ## CI report: * bbf0bc3e167ac3aa30f1044a352f4bdac4a416a2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=923) * bd7ca0b4407e126739f17292b8a6a0561508351e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=935) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KurtYoung commented on a change in pull request #11297: [FLINK-16364] [table] Deprecate the methods in TableEnvironment proposed by FLIP-84
KurtYoung commented on a change in pull request #11297:
URL: https://github.com/apache/flink/pull/11297#discussion_r422772319
##
File path:
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/utils/TableEnvUtil.scala
##
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.runtime.utils
Review comment:
It don't seem like a runtime util.
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/TableEnvironment.java
##
@@ -839,7 +845,9 @@ default Table fromValues(DataType rowType, Object...
values) {
*
* @param extended if the plan should contain additional properties,
* e.g. estimated cost, traits
+* @deprecated use {@link StatementSet#explain(ExplainDetail...)}.
*/
+ @Deprecated
Review comment:
We should also deprecate Line#830 and Line#840?
##
File path:
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/api/TableEnvironmentITCase.scala
##
@@ -26,7 +26,7 @@ import
org.apache.flink.table.api.internal.TableEnvironmentImpl
import org.apache.flink.table.api.java.StreamTableEnvironment
import org.apache.flink.table.api.scala.{StreamTableEnvironment =>
ScalaStreamTableEnvironment, _}
import
org.apache.flink.table.planner.factories.utils.TestCollectionTableFactory
-import org.apache.flink.table.planner.runtime.utils.TestingAppendSink
+import org.apache.flink.table.planner.runtime.utils.{TableEnvUtil,
TestingAppendSink}
Review comment:
There are still quite some deprecated method call in this test class, is
it intended?
##
File path:
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/utils/TableEnvUtil.scala
##
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.runtime.utils
+
+import org.apache.flink.api.common.JobExecutionResult
+import org.apache.flink.table.api.{Table, TableEnvironment}
+
+object TableEnvUtil {
+
+ def syncExecuteInsert(tEnv: TableEnvironment, insert: String):
JobExecutionResult = {
Review comment:
execInsertSqlAndWaitResult? Sync doesn't have clean semantic here.
##
File path:
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/utils/TableEnvUtil.scala
##
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.runtime.utils
+
+import org.apache.flink.api.common.JobExecutionResult
+import
[GitHub] [flink] flinkbot edited a comment on pull request #12065: [FLINK-17604] Implement format factory for CSV serialization and
flinkbot edited a comment on pull request #12065: URL: https://github.com/apache/flink/pull/12065#issuecomment-626450869 ## CI report: * a2a08d561137d48bdff518491863b331765df2a9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=931) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12061: [FLINK-17454][python]python gateway callback server port should not b…
flinkbot edited a comment on pull request #12061: URL: https://github.com/apache/flink/pull/12061#issuecomment-626432764 ## CI report: * d07c04b5ac978764112af3bf1d35285195e3440d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=924) * da1bf1e181793e94b1c9147546e15101e87e066f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12049: [FLINK-17252] [table] supports SELECT statement in TableEnvironment#executeSql and introduce Table#execute api
flinkbot edited a comment on pull request #12049: URL: https://github.com/apache/flink/pull/12049#issuecomment-626106371 ## CI report: * 3640d760380b6b527a115e09363f596fd0134c82 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=933) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12063: [FLINK-17603][table][core] Prepare Hive partitioned streaming source
flinkbot edited a comment on pull request #12063: URL: https://github.com/apache/flink/pull/12063#issuecomment-626441217 ## CI report: * 11d3939cde0deeac80e4642bb946622e7ee4d850 UNKNOWN * 64ffbab011482e857ba986b2116d9dfb94138259 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=928) * b809090fe19a17f34bbf25d20e4ad9bb5a618028 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12037: [FLINK-14807][runtime][client] Introduce communication through REST API between the clients and the operator coordinators
flinkbot edited a comment on pull request #12037: URL: https://github.com/apache/flink/pull/12037#issuecomment-625870801 ## CI report: * c06e5db9e64a2b39d857babff53d401697b87262 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=858) * 8317107cba32a7102c19e920b2a9932e40bbc5a7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=932) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
flinkbot edited a comment on pull request #12004: URL: https://github.com/apache/flink/pull/12004#issuecomment-624445192 ## CI report: * 44df52a0ff62e1b4ec65efd6a14a8b07d7712e75 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=872) * 3ac79a41395792dfecefc87a19c58468e91ede8d UNKNOWN * b3802f894dcdd7226316583e66b37a662c3e5bdb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=929) * 6545efc0050c0d2ae84755199cf3b5fd55e82dca UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11297: [FLINK-16364] [table] Deprecate the methods in TableEnvironment proposed by FLIP-84
flinkbot edited a comment on pull request #11297: URL: https://github.com/apache/flink/pull/11297#issuecomment-593931358 ## CI report: * bbf0bc3e167ac3aa30f1044a352f4bdac4a416a2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=923) * bd7ca0b4407e126739f17292b8a6a0561508351e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #11191: [FLINK-16094][docs] Translate /dev/table/functions/udfs.zh.md
xintongsong commented on pull request #11191: URL: https://github.com/apache/flink/pull/11191#issuecomment-626457779 Thanks @libenchao, I'm reviewing this right now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #11755: [FLINK-14257][Connectors / FileSystem]Integrate csv to FileSystemTableFactory
JingsongLi commented on a change in pull request #11755:
URL: https://github.com/apache/flink/pull/11755#discussion_r422441201
##
File path:
flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvRowDataFileSystemFormatFactory.java
##
@@ -0,0 +1,354 @@
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.io.InputFormat;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.GenericTypeInfo;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.core.fs.FileInputSplit;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.descriptors.DescriptorProperties;
+import org.apache.flink.table.factories.FileSystemFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.utils.PartitionPathUtils;
+
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_ALLOW_COMMENTS;
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_ARRAY_ELEMENT_DELIMITER;
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_DISABLE_QUOTE_CHARACTER;
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_ESCAPE_CHARACTER;
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_FIELD_DELIMITER;
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_IGNORE_PARSE_ERRORS;
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_LINE_DELIMITER;
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_NULL_LITERAL;
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_QUOTE_CHARACTER;
+import static
org.apache.flink.table.descriptors.FormatDescriptorValidator.FORMAT;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * CSV format factory for file system.
+ */
+public class CsvRowDataFileSystemFormatFactory implements
FileSystemFormatFactory {
+ @Override
Review comment:
Add empty line.
##
File path:
flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvRowDataFileSystemFormatFactory.java
##
@@ -0,0 +1,354 @@
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.io.InputFormat;
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.GenericTypeInfo;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.core.fs.FileInputSplit;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.descriptors.DescriptorProperties;
+import org.apache.flink.table.factories.FileSystemFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.utils.PartitionPathUtils;
+
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_ALLOW_COMMENTS;
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_ARRAY_ELEMENT_DELIMITER;
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_DISABLE_QUOTE_CHARACTER;
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_ESCAPE_CHARACTER;
+import static
org.apache.flink.table.descriptors.CsvValidator.FORMAT_FIELD_DELIMITER;
+import static
[jira] [Closed] (FLINK-17601) TableEnvironmentITCase.testStatementSet fails on travis
[
https://issues.apache.org/jira/browse/FLINK-17601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young closed FLINK-17601.
--
Resolution: Fixed
master: 51c7d61f3e7661f6aad331245ca94ecb82be3994
> TableEnvironmentITCase.testStatementSet fails on travis
> ---
>
> Key: FLINK-17601
> URL: https://issues.apache.org/jira/browse/FLINK-17601
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: Dawid Wysakowicz
>Assignee: godfrey he
>Priority: Blocker
> Fix For: 1.11.0
>
>
> {code}
> [ERROR] Failures:
> [ERROR] TableEnvironmentITCase.testStatementSet:446 expected:<...
> fields=[first])
> +- [TableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- ]TableSourceScan(tabl...> but was:<... fields=[first])
> +- [LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- Legacy]TableSourceScan(tabl...>
> [ERROR] TableEnvironmentITCase.testStatementSet:446 expected:<...
> fields=[first])
> +- [TableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- ]TableSourceScan(tabl...> but was:<... fields=[first])
> +- [LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- Legacy]TableSourceScan(tabl...>
> [ERROR] TableEnvironmentITCase.testStatementSet:446 expected:<...
> fields=[first])
> +- [TableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- ]TableSourceScan(tabl...> but was:<... fields=[first])
> +- [LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- Legacy]TableSourceScan(tabl...>
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=912=logs=e25d5e7e-2a9c-5589-4940-0b638d75a414=f83cd372-208c-5ec4-12a8-337462457129
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] shuiqiangchen commented on a change in pull request #12061: [FLINK-17454][python]python gateway callback server port should not b…
shuiqiangchen commented on a change in pull request #12061: URL: https://github.com/apache/flink/pull/12061#discussion_r422764493 ## File path: flink-python/pyflink/java_gateway.py ## @@ -49,15 +49,19 @@ def get_gateway(): # if Java Gateway is already running if 'PYFLINK_GATEWAY_PORT' in os.environ: gateway_port = int(os.environ['PYFLINK_GATEWAY_PORT']) -callback_port = int(os.environ['PYFLINK_CALLBACK_PORT']) gateway_param = GatewayParameters(port=gateway_port, auto_convert=True) _gateway = JavaGateway( gateway_parameters=gateway_param, callback_server_parameters=CallbackServerParameters( -port=callback_port, daemonize=True, daemonize_connections=True)) +port=0, daemonize=True, daemonize_connections=True)) else: _gateway = launch_gateway() +callback_server = _gateway.get_callback_server() +listening_callback_address = callback_server.get_listening_address() Review comment: Thanks, I'll rename it to be more intuitive. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on a change in pull request #12061: [FLINK-17454][python]python gateway callback server port should not b…
shuiqiangchen commented on a change in pull request #12061:
URL: https://github.com/apache/flink/pull/12061#discussion_r422764356
##
File path:
flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java
##
@@ -300,6 +294,48 @@ static GatewayServer startGatewayServer() throws
ExecutionException, Interrupted
return gatewayServerFuture.get();
}
+ /**
+* Reset a daemon thread to the callback client thread pool so that the
callback server can be terminated when gate
+* way server is shutting down. We need to shut down the none-daemon
thread firstly, then set a new thread created
+* in a daemon thread to the ExecutorService.
+*
+* @param gatewayServer the gateway which creates the callback server.
+* */
+ private static void resetCallbackClientExecutorService(GatewayServer
gatewayServer) throws NoSuchFieldException,
+ IllegalAccessException, NoSuchMethodException,
InvocationTargetException {
+ CallbackClient callbackClient = (CallbackClient)
gatewayServer.getCallbackClient();
+ // The Java API of py4j does not provide approach to set
"daemonize_connections" parameter.
+ // Use reflect to daemonize the connection thread.
+ Field executor =
CallbackClient.class.getDeclaredField("executor");
+ executor.setAccessible(true);
+ ((ScheduledExecutorService)
executor.get(callbackClient)).shutdown();
+ executor.set(callbackClient,
Executors.newScheduledThreadPool(1, Thread::new));
+ Method setupCleaner =
CallbackClient.class.getDeclaredMethod("setupCleaner");
+ setupCleaner.setAccessible(true);
+ setupCleaner.invoke(callbackClient);
+ }
+
+ /**
+* Reset the callback client of gatewayServer with the given
callbackListeningAddress and callbackListeningPort
+* after the callback server started.
+*
+* @param callbackListeningAddress the listening address of the
callback server.
+* @param callbackListeningPort the listening port of the callback
server.
+* */
+ public static void resetCallbackClient(String callbackListeningAddress,
int callbackListeningPort) throws
+ UnknownHostException, InvocationTargetException,
NoSuchMethodException, IllegalAccessException,
+ NoSuchFieldException {
+
+ gatewayServer = getGatewayServer();
+ if (gatewayServer == null){
Review comment:
Yes, there is no need do this judgement I'll remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-17598) Implement FileSystemHAServices for native K8s setups
[
https://issues.apache.org/jira/browse/FLINK-17598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104040#comment-17104040
]
Canbin Zheng edited comment on FLINK-17598 at 5/11/20, 3:48 AM:
[~fly_in_gis] Not exactly. The *StatefulSet* could ensure at-most-one semantics
for each individual pod. *StatefulSet* have strong guarantees regarding the
existence of the pods. Pods are uniquely identified with an ordinal index.
These pods are guaranteed to be spun up in ascending order of their index and
taken down in descending order.
Here is one of the most important features of *StatefulSet* regarding failover:
*StatefulSet* have a guarantee that there will never be more than 1 instance
of a pod at any given time, which is different from a deployment.
was (Author: felixzheng):
[~fly_in_gis] Not exactly. The *StatefulSet* could ensure at-most-one semantics
for each individual pod. *Statefulsets* have strong guarantees regarding the
existence of the pods. Pods are uniquely identified with an ordinal index.
These pods are guaranteed to be spun up in ascending order of their index and
taken down in descending order.
Here is one of the most important features of *StatefulSet* regarding failover:
*StatefulSets* have a guarantee that there will never be more than 1 instance
of a pod at any given time, which is different from a deployment.
> Implement FileSystemHAServices for native K8s setups
>
>
> Key: FLINK-17598
> URL: https://issues.apache.org/jira/browse/FLINK-17598
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / Kubernetes, Runtime / Coordination
>Reporter: Canbin Zheng
>Priority: Major
>
> At the moment we use Zookeeper as a distributed coordinator for implementing
> JobManager high availability services. But in the cloud-native environment,
> there is a trend that more and more users prefer to use *Kubernetes* as the
> underlying scheduler backend while *Storage Object* as the Storage medium,
> both of these two services don't require Zookeeper deployment.
> As a result, in the K8s setups, people have to deploy and maintain their
> Zookeeper clusters for solving JobManager SPOF. This ticket proposes to
> provide a simplified FileSystem HA implementation with the leader-election
> removed, which saves the efforts of Zookeeper deployment.
> To achieve this, we plan to
> # Introduce a {{FileSystemHaServices}} which implements the
> {{HighAvailabilityServices}}.
> # Replace Deployment with StatefulSet to ensure *at most one* semantics,
> preventing potential concurrent access to the underlying FileSystem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12049: [FLINK-17252] [table] supports SELECT statement in TableEnvironment#executeSql and introduce Table#execute api
flinkbot edited a comment on pull request #12049: URL: https://github.com/apache/flink/pull/12049#issuecomment-626106371 ## CI report: * 50dfd19d62da7a39f2004934703344b7f3d95573 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=921) * 3640d760380b6b527a115e09363f596fd0134c82 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12037: [FLINK-14807][runtime][client] Introduce communication through REST API between the clients and the operator coordinators
flinkbot edited a comment on pull request #12037: URL: https://github.com/apache/flink/pull/12037#issuecomment-625870801 ## CI report: * c06e5db9e64a2b39d857babff53d401697b87262 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=858) * 8317107cba32a7102c19e920b2a9932e40bbc5a7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12063: [FLINK-17603][table][core] Prepare Hive partitioned streaming source
flinkbot edited a comment on pull request #12063: URL: https://github.com/apache/flink/pull/12063#issuecomment-626441217 ## CI report: * 11d3939cde0deeac80e4642bb946622e7ee4d850 UNKNOWN * 64ffbab011482e857ba986b2116d9dfb94138259 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=928) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12010: [FLINK-17286][connectors / filesystem]Integrate json to file system connector
flinkbot edited a comment on pull request #12010: URL: https://github.com/apache/flink/pull/12010#issuecomment-624746999 ## CI report: * d7445e68ed77a6b1b88fdf0dac8638ff62ac8967 UNKNOWN * edacdf8f0062fdae68a1581936a7410c1b35baa2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=926) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17598) Implement FileSystemHAServices for native K8s setups
[
https://issues.apache.org/jira/browse/FLINK-17598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104040#comment-17104040
]
Canbin Zheng commented on FLINK-17598:
--
[~fly_in_gis] Not exactly. The *StatefulSet* could ensure at-most-one semantics
for each individual pod. *Statefulsets* have strong guarantees regarding the
existence of the pods. Pods are uniquely identified with an ordinal index.
These pods are guaranteed to be spun up in ascending order of their index and
taken down in descending order.
Here is one of the most important features of *StatefulSet* regarding failover:
*StatefulSets* have a guarantee that there will never be more than 1 instance
of a pod at any given time, which is different from a deployment.
> Implement FileSystemHAServices for native K8s setups
>
>
> Key: FLINK-17598
> URL: https://issues.apache.org/jira/browse/FLINK-17598
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / Kubernetes, Runtime / Coordination
>Reporter: Canbin Zheng
>Priority: Major
>
> At the moment we use Zookeeper as a distributed coordinator for implementing
> JobManager high availability services. But in the cloud-native environment,
> there is a trend that more and more users prefer to use *Kubernetes* as the
> underlying scheduler backend while *Storage Object* as the Storage medium,
> both of these two services don't require Zookeeper deployment.
> As a result, in the K8s setups, people have to deploy and maintain their
> Zookeeper clusters for solving JobManager SPOF. This ticket proposes to
> provide a simplified FileSystem HA implementation with the leader-election
> removed, which saves the efforts of Zookeeper deployment.
> To achieve this, we plan to
> # Introduce a {{FileSystemHaServices}} which implements the
> {{HighAvailabilityServices}}.
> # Replace Deployment with StatefulSet to ensure *at most one* semantics,
> preventing potential concurrent access to the underlying FileSystem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12065: [FLINK-17604] Implement format factory for CSV serialization and
flinkbot edited a comment on pull request #12065: URL: https://github.com/apache/flink/pull/12065#issuecomment-626450869 ## CI report: * a2a08d561137d48bdff518491863b331765df2a9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=931) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
flinkbot edited a comment on pull request #12004: URL: https://github.com/apache/flink/pull/12004#issuecomment-624445192 ## CI report: * 44df52a0ff62e1b4ec65efd6a14a8b07d7712e75 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=872) * 3ac79a41395792dfecefc87a19c58468e91ede8d UNKNOWN * b3802f894dcdd7226316583e66b37a662c3e5bdb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12064: [hotfix][javadocs]java doc error fix
flinkbot edited a comment on pull request #12064: URL: https://github.com/apache/flink/pull/12064#issuecomment-626450834 ## CI report: * a799713860db44d58407123f19236b68e231094b Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/165085794) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=930) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lirui-apache commented on a change in pull request #11935: [FLINK-17431][sql-parser-hive][hive] Implement table DDLs for Hive di…
lirui-apache commented on a change in pull request #11935:
URL: https://github.com/apache/flink/pull/11935#discussion_r422763697
##
File path:
flink-table/flink-sql-parser-hive/src/main/codegen/includes/parserImpls.ftl
##
@@ -218,4 +218,549 @@ SqlNode TableOption() :
{
return new SqlTableOption(key, value, getPos());
}
-}
\ No newline at end of file
+}
+
+
+SqlCreate SqlCreateTemporary(Span s, boolean replace) :
+{
+ boolean isTemporary = false;
+ SqlCreate create;
+}
+{
+ [{isTemporary = true;} ]
+
+ create = SqlCreateTable(s, isTemporary)
+ {
+return create;
+ }
+}
+
+/**
+* Parse a "Show Tables" metadata query command.
+*/
+SqlShowTables SqlShowTables() :
+{
+}
+{
+
+{
+return new SqlShowTables(getPos());
+}
+}
+
+/**
+ * Here we add Rich in className to distinguish from calcite's original
SqlDescribeTable.
+ */
+SqlRichDescribeTable SqlRichDescribeTable() :
+{
+SqlIdentifier tableName;
+SqlParserPos pos;
+boolean isExtended = false;
+}
+{
+ { pos = getPos();}
+[ LOOKAHEAD(2) ( | ) { isExtended = true;} ]
+tableName = CompoundIdentifier()
+{
+return new SqlRichDescribeTable(pos, tableName, isExtended);
+}
+}
+
+SqlCreate SqlCreateTable(Span s, boolean isTemporary) :
+{
+final SqlParserPos startPos = s.pos();
+SqlIdentifier tableName;
+SqlNodeList primaryKeyList = SqlNodeList.EMPTY;
+List uniqueKeysList = new ArrayList();
+SqlNodeList columnList = SqlNodeList.EMPTY;
+ SqlCharStringLiteral comment = null;
+
+SqlNodeList propertyList;
+SqlNodeList partitionColumns = SqlNodeList.EMPTY;
+SqlParserPos pos = startPos;
+boolean isExternal = false;
+HiveTableRowFormat rowFormat = null;
+HiveTableStoredAs storedAs = null;
+SqlCharStringLiteral location = null;
+HiveTableCreationContext ctx = new HiveTableCreationContext();
+}
+{
+[ { isExternal = true; } ]
+ { propertyList = new SqlNodeList(getPos()); }
+
+tableName = CompoundIdentifier()
+[
+ { pos = getPos(); }
+TableColumn(ctx)
+(
+ TableColumn(ctx)
+)*
+{
+pos = pos.plus(getPos());
+columnList = new SqlNodeList(ctx.columnList, pos);
+}
+
+]
+[ {
+comment = createStringLiteral(token.image, getPos());
+}]
+[
+
+
+ {
+List partCols = new ArrayList();
+if ( columnList == SqlNodeList.EMPTY ) {
+ columnList = new SqlNodeList(pos.plus(getPos()));
+}
+ }
+ PartColumnDef(partCols)
+ (
+ PartColumnDef(partCols)
+ )*
+ {
+partitionColumns = new SqlNodeList(partCols, pos.plus(getPos()));
+ }
+
+]
+[
+
+ rowFormat = TableRowFormat(getPos())
+]
+[
+
+ storedAs = TableStoredAs(getPos())
+]
+[
+
+ { location = createStringLiteral(token.image, getPos()); }
+]
+[
+
+{
+ SqlNodeList props = TableProperties();
+ for (SqlNode node : props) {
+propertyList.add(node);
+ }
+}
+]
+{
+return new SqlCreateHiveTable(startPos.plus(getPos()),
+tableName,
+columnList,
+ctx,
+propertyList,
+partitionColumns,
+comment,
+isTemporary,
+isExternal,
+rowFormat,
+storedAs,
+location);
+}
+}
+
+SqlDrop SqlDropTable(Span s, boolean replace) :
+{
+SqlIdentifier tableName = null;
+boolean ifExists = false;
+}
+{
+
+
+(
+ { ifExists = true; }
+|
+{ ifExists = false; }
+)
+
+tableName = CompoundIdentifier()
+
+{
+ return new SqlDropTable(s.pos(), tableName, ifExists, false);
+}
+}
+
+void TableColumn2(List list) :
+{
+SqlParserPos pos;
+SqlIdentifier name;
+SqlDataTypeSpec type;
+SqlCharStringLiteral comment = null;
+}
+{
+name = SimpleIdentifier()
+type = ExtendedDataType()
+[ {
+comment = createStringLiteral(token.image, getPos());
+}]
+{
+SqlTableColumn tableColumn = new SqlTableColumn(name, type, null,
comment, getPos());
+list.add(tableColumn);
+}
+}
+
+void PartColumnDef(List list) :
+{
+SqlParserPos pos;
+SqlIdentifier name;
+SqlDataTypeSpec type;
+SqlCharStringLiteral comment = null;
+}
+{
+name = SimpleIdentifier()
+type = DataType()
+[ {
+comment = createStringLiteral(token.image, getPos());
+}]
+{
+type = type.withNullable(true);
+SqlTableColumn tableColumn = new SqlTableColumn(name, type, null,
comment, getPos());
+list.add(tableColumn);
+}
+}
+
+void TableColumn(HiveTableCreationContext context) :
+{
+}
[GitHub] [flink] danny0405 commented on a change in pull request #11935: [FLINK-17431][sql-parser-hive][hive] Implement table DDLs for Hive di…
danny0405 commented on a change in pull request #11935:
URL: https://github.com/apache/flink/pull/11935#discussion_r422761906
##
File path:
flink-table/flink-sql-parser-hive/src/main/codegen/includes/parserImpls.ftl
##
@@ -218,4 +218,549 @@ SqlNode TableOption() :
{
return new SqlTableOption(key, value, getPos());
}
-}
\ No newline at end of file
+}
+
+
+SqlCreate SqlCreateTemporary(Span s, boolean replace) :
+{
+ boolean isTemporary = false;
+ SqlCreate create;
+}
+{
+ [{isTemporary = true;} ]
+
+ create = SqlCreateTable(s, isTemporary)
+ {
+return create;
+ }
+}
+
+/**
+* Parse a "Show Tables" metadata query command.
+*/
+SqlShowTables SqlShowTables() :
+{
+}
+{
+
+{
+return new SqlShowTables(getPos());
+}
+}
+
+/**
+ * Here we add Rich in className to distinguish from calcite's original
SqlDescribeTable.
+ */
+SqlRichDescribeTable SqlRichDescribeTable() :
+{
+SqlIdentifier tableName;
+SqlParserPos pos;
+boolean isExtended = false;
+}
+{
+ { pos = getPos();}
+[ LOOKAHEAD(2) ( | ) { isExtended = true;} ]
+tableName = CompoundIdentifier()
+{
+return new SqlRichDescribeTable(pos, tableName, isExtended);
+}
+}
+
+SqlCreate SqlCreateTable(Span s, boolean isTemporary) :
+{
+final SqlParserPos startPos = s.pos();
+SqlIdentifier tableName;
+SqlNodeList primaryKeyList = SqlNodeList.EMPTY;
+List uniqueKeysList = new ArrayList();
+SqlNodeList columnList = SqlNodeList.EMPTY;
+ SqlCharStringLiteral comment = null;
+
+SqlNodeList propertyList;
+SqlNodeList partitionColumns = SqlNodeList.EMPTY;
+SqlParserPos pos = startPos;
+boolean isExternal = false;
+HiveTableRowFormat rowFormat = null;
+HiveTableStoredAs storedAs = null;
+SqlCharStringLiteral location = null;
+HiveTableCreationContext ctx = new HiveTableCreationContext();
+}
+{
+[ { isExternal = true; } ]
+ { propertyList = new SqlNodeList(getPos()); }
+
+tableName = CompoundIdentifier()
+[
+ { pos = getPos(); }
+TableColumn(ctx)
+(
+ TableColumn(ctx)
+)*
+{
+pos = pos.plus(getPos());
+columnList = new SqlNodeList(ctx.columnList, pos);
+}
+
+]
+[ {
+comment = createStringLiteral(token.image, getPos());
+}]
+[
+
+
+ {
+List partCols = new ArrayList();
+if ( columnList == SqlNodeList.EMPTY ) {
+ columnList = new SqlNodeList(pos.plus(getPos()));
+}
+ }
+ PartColumnDef(partCols)
+ (
+ PartColumnDef(partCols)
+ )*
+ {
+partitionColumns = new SqlNodeList(partCols, pos.plus(getPos()));
+ }
+
+]
+[
+
+ rowFormat = TableRowFormat(getPos())
+]
+[
+
+ storedAs = TableStoredAs(getPos())
+]
+[
+
+ { location = createStringLiteral(token.image, getPos()); }
+]
+[
+
+{
+ SqlNodeList props = TableProperties();
+ for (SqlNode node : props) {
+propertyList.add(node);
+ }
+}
+]
+{
+return new SqlCreateHiveTable(startPos.plus(getPos()),
+tableName,
+columnList,
+ctx,
+propertyList,
+partitionColumns,
+comment,
+isTemporary,
+isExternal,
+rowFormat,
+storedAs,
+location);
+}
+}
+
+SqlDrop SqlDropTable(Span s, boolean replace) :
+{
+SqlIdentifier tableName = null;
+boolean ifExists = false;
+}
+{
+
+
+(
+ { ifExists = true; }
+|
+{ ifExists = false; }
+)
+
+tableName = CompoundIdentifier()
+
+{
+ return new SqlDropTable(s.pos(), tableName, ifExists, false);
+}
+}
+
+void TableColumn2(List list) :
+{
+SqlParserPos pos;
+SqlIdentifier name;
+SqlDataTypeSpec type;
+SqlCharStringLiteral comment = null;
+}
+{
+name = SimpleIdentifier()
+type = ExtendedDataType()
+[ {
+comment = createStringLiteral(token.image, getPos());
+}]
+{
+SqlTableColumn tableColumn = new SqlTableColumn(name, type, null,
comment, getPos());
+list.add(tableColumn);
+}
+}
+
+void PartColumnDef(List list) :
+{
+SqlParserPos pos;
+SqlIdentifier name;
+SqlDataTypeSpec type;
+SqlCharStringLiteral comment = null;
+}
+{
+name = SimpleIdentifier()
+type = DataType()
+[ {
+comment = createStringLiteral(token.image, getPos());
+}]
+{
+type = type.withNullable(true);
+SqlTableColumn tableColumn = new SqlTableColumn(name, type, null,
comment, getPos());
+list.add(tableColumn);
+}
+}
+
+void TableColumn(HiveTableCreationContext context) :
+{
+}
+{
+
[jira] [Commented] (FLINK-17598) Implement FileSystemHAServices for native K8s setups
[
https://issues.apache.org/jira/browse/FLINK-17598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104036#comment-17104036
]
Yang Wang commented on FLINK-17598:
---
Hi [~felixzheng], if we do not have the leader election, i think there is no
guarantee that only one active jobmanager is running. For example, kubelet is
down and jobmanager pod is still running, a new jobmanager pod will be
launched. Then we will have two jobmanager running and read/write the meta
store, it will cause some unrecoverable issues.
> Implement FileSystemHAServices for native K8s setups
>
>
> Key: FLINK-17598
> URL: https://issues.apache.org/jira/browse/FLINK-17598
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / Kubernetes, Runtime / Coordination
>Reporter: Canbin Zheng
>Priority: Major
>
> At the moment we use Zookeeper as a distributed coordinator for implementing
> JobManager high availability services. But in the cloud-native environment,
> there is a trend that more and more users prefer to use *Kubernetes* as the
> underlying scheduler backend while *Storage Object* as the Storage medium,
> both of these two services don't require Zookeeper deployment.
> As a result, in the K8s setups, people have to deploy and maintain their
> Zookeeper clusters for solving JobManager SPOF. This ticket proposes to
> provide a simplified FileSystem HA implementation with the leader-election
> removed, which saves the efforts of Zookeeper deployment.
> To achieve this, we plan to
> # Introduce a {{FileSystemHaServices}} which implements the
> {{HighAvailabilityServices}}.
> # Replace Deployment with StatefulSet to ensure *at most one* semantics,
> preventing potential concurrent access to the underlying FileSystem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17416) Flink-kubernetes doesn't work on java 8 8u252
[ https://issues.apache.org/jira/browse/FLINK-17416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104034#comment-17104034 ] Yang Wang commented on FLINK-17416: --- Some updates on this ticket. After more investigation and testing, i find that this issue is not directly caused by the bug of okhttp. Instead, it is related with both kubernetes version and jdk version. * For jdk 8u252, the fabric8 kubernetes-client could only work on kubernetes v1.16 and lower versions. * For other jdk versions(e.g. 8u242, jdk11), i am not aware of the same issues. It seems that native K8s integration works well. The user in the ML have solved the problem after upgrading to jdk11. I will keep in touch with the author of fabric8 kubernetes-client to verify when this could be resolved. At the same time i will update the K8s e2e tests to use kubernetes version v1.16.9 and enable the e2e tests so that it will not block our general release cycle. [~rmetzger] What do you think? > Flink-kubernetes doesn't work on java 8 8u252 > - > > Key: FLINK-17416 > URL: https://issues.apache.org/jira/browse/FLINK-17416 > Project: Flink > Issue Type: Bug > Components: Deployment / Kubernetes >Affects Versions: 1.10.0, 1.11.0 >Reporter: wangxiyuan >Priority: Blocker > Labels: pull-request-available > Fix For: 1.11.0 > > Attachments: log.k8s.session.8u252 > > > When using java-8-8u252 version, the flink container end-to-end failed. The > test `Running 'Run kubernetes session test'` fails with the `Broken pipe` > error. > See: > [https://logs.openlabtesting.org/logs/periodic-20-flink-mail/github.com/apache/flink/master/flink-end-to-end-test-arm64-container/fcfdd47/job-output.txt.gz] > > Flink Azure CI doesn't hit this problem because it runs under jdk-8-8u242 > > The reason is that the okhttp library which flink using doesn't work on > java-8-8u252: > [https://github.com/square/okhttp/issues/5970] > > The problem has been with the PR: > [https://github.com/square/okhttp/pull/5977] > > Maybe we can wait for a new 3.12.x release and bump the okhttp version in > Flink later. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12064: [hotfix][javadocs]java doc error fix
flinkbot commented on pull request #12064: URL: https://github.com/apache/flink/pull/12064#issuecomment-626450834 ## CI report: * a799713860db44d58407123f19236b68e231094b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12065: [FLINK-17604] Implement format factory for CSV serialization and
flinkbot commented on pull request #12065: URL: https://github.com/apache/flink/pull/12065#issuecomment-626450869 ## CI report: * a2a08d561137d48bdff518491863b331765df2a9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12063: [FLINK-17603][table][core] Prepare Hive partitioned streaming source
flinkbot edited a comment on pull request #12063: URL: https://github.com/apache/flink/pull/12063#issuecomment-626441217 ## CI report: * 11d3939cde0deeac80e4642bb946622e7ee4d850 UNKNOWN * 64ffbab011482e857ba986b2116d9dfb94138259 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=928) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
flinkbot edited a comment on pull request #12004: URL: https://github.com/apache/flink/pull/12004#issuecomment-624445192 ## CI report: * 44df52a0ff62e1b4ec65efd6a14a8b07d7712e75 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=872) * 3ac79a41395792dfecefc87a19c58468e91ede8d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17601) TableEnvironmentITCase.testStatementSet fails on travis
[
https://issues.apache.org/jira/browse/FLINK-17601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104030#comment-17104030
]
godfrey he commented on FLINK-17601:
thanks for reporting this [~dwysakowicz], I have opened a pr
https://github.com/apache/flink/pull/12060
> TableEnvironmentITCase.testStatementSet fails on travis
> ---
>
> Key: FLINK-17601
> URL: https://issues.apache.org/jira/browse/FLINK-17601
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: Dawid Wysakowicz
>Assignee: godfrey he
>Priority: Blocker
> Fix For: 1.11.0
>
>
> {code}
> [ERROR] Failures:
> [ERROR] TableEnvironmentITCase.testStatementSet:446 expected:<...
> fields=[first])
> +- [TableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- ]TableSourceScan(tabl...> but was:<... fields=[first])
> +- [LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- Legacy]TableSourceScan(tabl...>
> [ERROR] TableEnvironmentITCase.testStatementSet:446 expected:<...
> fields=[first])
> +- [TableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- ]TableSourceScan(tabl...> but was:<... fields=[first])
> +- [LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- Legacy]TableSourceScan(tabl...>
> [ERROR] TableEnvironmentITCase.testStatementSet:446 expected:<...
> fields=[first])
> +- [TableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- ]TableSourceScan(tabl...> but was:<... fields=[first])
> +- [LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- Legacy]TableSourceScan(tabl...>
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=912=logs=e25d5e7e-2a9c-5589-4940-0b638d75a414=f83cd372-208c-5ec4-12a8-337462457129
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
JingsongLi commented on a change in pull request #12004:
URL: https://github.com/apache/flink/pull/12004#discussion_r422757793
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/TimestampedHiveInputSplit.java
##
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.hive.read;
+
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInfoFactory;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.api.java.typeutils.runtime.DataInputViewStream;
+import org.apache.flink.api.java.typeutils.runtime.DataOutputViewStream;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.streaming.api.functions.source.TimestampedInputSplit;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.lang.reflect.Type;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * A {@link HiveTableInputSplit} with {@link TimestampedInputSplit}.
+ * Kryo serializer can not deal with hadoop split, need specific type
information factory.
+ */
+@TypeInfo(TimestampedHiveInputSplit.SplitTypeInfoFactory.class)
+public class TimestampedHiveInputSplit extends HiveTableInputSplit implements
TimestampedInputSplit {
+
+ private static final long serialVersionUID = 1L;
+
+ /** The modification time of the file this split belongs to. */
+ private final long modificationTime;
+
+ /**
+* The state of the split. This information is used when
+* restoring from a checkpoint and allows to resume reading the
+* underlying file from the point we left off.
+* */
+ private Serializable splitState;
+
+ public TimestampedHiveInputSplit(
+ long modificationTime,
+ HiveTableInputSplit split) {
+ super(
+ split.getSplitNumber(),
+ split.getHadoopInputSplit(),
+ split.getJobConf(),
+ split.getHiveTablePartition());
+ this.modificationTime = modificationTime;
+ }
+
+ @Override
+ public void setSplitState(Serializable state) {
+ this.splitState = state;
+ }
+
+ @Override
+ public Serializable getSplitState() {
+ return this.splitState;
+ }
+
+ @Override
+ public long getModificationTime() {
+ return modificationTime;
+ }
+
+ @Override
+ public int compareTo(TimestampedInputSplit o) {
+ TimestampedHiveInputSplit split = (TimestampedHiveInputSplit) o;
+ int modTimeComp = Long.compare(this.modificationTime,
split.modificationTime);
+ if (modTimeComp != 0L) {
+ return modTimeComp;
+ }
+
+ int pathComp =
this.hiveTablePartition.getStorageDescriptor().compareTo(
+
split.hiveTablePartition.getStorageDescriptor());
+
+ return pathComp != 0 ? pathComp :
+ this.getSplitNumber() - o.getSplitNumber();
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ if (!super.equals(o)) {
Review comment:
`TimestampedHiveInputSplit` is only used in
`ContinuousFileReaderOperator`. I will add comments.
This is
[GitHub] [flink] TsReaper commented on a change in pull request #12037: [FLINK-14807][runtime][client] Introduce communication through REST API between the clients and the operator coordinators
TsReaper commented on a change in pull request #12037:
URL: https://github.com/apache/flink/pull/12037#discussion_r422757580
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/CoordinationResponser.java
##
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.operators.coordination;
+
+/**
+ * Coordinator interface which can handle {@link CoordinationRequest}s
+ * and response with {@link CoordinationResponse}s to the client.
+ */
+public interface CoordinationResponser {
Review comment:
`CoordinationRequestHandler` seems to be better
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12065: [FLINK-17604] Implement format factory for CSV serialization and
flinkbot commented on pull request #12065: URL: https://github.com/apache/flink/pull/12065#issuecomment-626447843 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit a2a08d561137d48bdff518491863b331765df2a9 (Mon May 11 03:15:34 UTC 2020) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #12065: [FLINK-17604] Implement format factory for CSV serialization and
JingsongLi commented on a change in pull request #12065:
URL: https://github.com/apache/flink/pull/12065#discussion_r422757205
##
File path:
flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatFactory.java
##
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.csv;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.ScanFormat;
+import org.apache.flink.table.connector.format.SinkFormat;
+import org.apache.flink.table.connector.source.ScanTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * Format factory for providing configured instances of CSV to RowData {@link
SerializationSchema}
+ * and {@link DeserializationSchema}.
+ */
+public final class CsvFormatFactory implements
+ DeserializationFormatFactory,
+ SerializationFormatFactory {
+
+ public static final String IDENTIFIER = "csv";
+
+ //
+ // Options
+ //
+
+ private static final ConfigOption FIELD_DELIMITER =
ConfigOptions
Review comment:
Can you extract these config options to a `CsvOptions`? Because file
system also requires these.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12064: [hotfix][javadocs]java doc error fix
flinkbot commented on pull request #12064: URL: https://github.com/apache/flink/pull/12064#issuecomment-626447459 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit a799713860db44d58407123f19236b68e231094b (Mon May 11 03:13:37 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17604) Implement format factory for CSV serialization and deseriazation schema of RowData type
[ https://issues.apache.org/jira/browse/FLINK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17604: --- Labels: pull-request-available (was: ) > Implement format factory for CSV serialization and deseriazation schema of > RowData type > --- > > Key: FLINK-17604 > URL: https://issues.apache.org/jira/browse/FLINK-17604 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.11.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] danny0405 opened a new pull request #12065: [FLINK-17604] Implement format factory for CSV serialization and
danny0405 opened a new pull request #12065: URL: https://github.com/apache/flink/pull/12065 deseriazation schema of RowData type ## What is the purpose of the change Implements format factory for CSV RowData Se/De schema so that they can use in SQL. ## Brief change log - A new CsvFormatFactory, the options are in style of what FLIP-122 has concluded. ## Verifying this change Added UTs. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? not documented This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhenxianyimeng opened a new pull request #12064: [hotfix][javadocs]java doc error fix
zhenxianyimeng opened a new pull request #12064: URL: https://github.com/apache/flink/pull/12064 ## What is the purpose of the change java doc in ExecutionEnvironment.readTextFile is error ## Brief change log change Default is UTF-0 to UTF-8 ## Verifying this change ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lirui-apache commented on a change in pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
lirui-apache commented on a change in pull request #12004:
URL: https://github.com/apache/flink/pull/12004#discussion_r422756417
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/TimestampedHiveInputSplit.java
##
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.hive.read;
+
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInfoFactory;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.api.java.typeutils.runtime.DataInputViewStream;
+import org.apache.flink.api.java.typeutils.runtime.DataOutputViewStream;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.streaming.api.functions.source.TimestampedInputSplit;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.lang.reflect.Type;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * A {@link HiveTableInputSplit} with {@link TimestampedInputSplit}.
+ * Kryo serializer can not deal with hadoop split, need specific type
information factory.
+ */
+@TypeInfo(TimestampedHiveInputSplit.SplitTypeInfoFactory.class)
+public class TimestampedHiveInputSplit extends HiveTableInputSplit implements
TimestampedInputSplit {
+
+ private static final long serialVersionUID = 1L;
+
+ /** The modification time of the file this split belongs to. */
+ private final long modificationTime;
+
+ /**
+* The state of the split. This information is used when
+* restoring from a checkpoint and allows to resume reading the
+* underlying file from the point we left off.
+* */
+ private Serializable splitState;
+
+ public TimestampedHiveInputSplit(
+ long modificationTime,
+ HiveTableInputSplit split) {
+ super(
+ split.getSplitNumber(),
+ split.getHadoopInputSplit(),
+ split.getJobConf(),
+ split.getHiveTablePartition());
+ this.modificationTime = modificationTime;
+ }
+
+ @Override
+ public void setSplitState(Serializable state) {
+ this.splitState = state;
+ }
+
+ @Override
+ public Serializable getSplitState() {
+ return this.splitState;
+ }
+
+ @Override
+ public long getModificationTime() {
+ return modificationTime;
+ }
+
+ @Override
+ public int compareTo(TimestampedInputSplit o) {
+ TimestampedHiveInputSplit split = (TimestampedHiveInputSplit) o;
+ int modTimeComp = Long.compare(this.modificationTime,
split.modificationTime);
+ if (modTimeComp != 0L) {
+ return modTimeComp;
+ }
+
+ int pathComp =
this.hiveTablePartition.getStorageDescriptor().compareTo(
+
split.hiveTablePartition.getStorageDescriptor());
+
+ return pathComp != 0 ? pathComp :
+ this.getSplitNumber() - o.getSplitNumber();
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ if (!super.equals(o)) {
Review comment:
I think the best practice is to avoid such implementations. And if for
some reason it has to be done this way, we should clarify it in the java doc,
according to the
[GitHub] [flink] flinkbot edited a comment on pull request #12062: [FLINK-17587][filesystem] Filesystem streaming sink support commit success file
flinkbot edited a comment on pull request #12062: URL: https://github.com/apache/flink/pull/12062#issuecomment-626437386 ## CI report: * 38d4e669261a6fde7595223bf804b45e838476b2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=927) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12061: [FLINK-17454][python]python gateway callback server port should not b…
flinkbot edited a comment on pull request #12061: URL: https://github.com/apache/flink/pull/12061#issuecomment-626432764 ## CI report: * d07c04b5ac978764112af3bf1d35285195e3440d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=924) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #12037: [FLINK-14807][runtime][client] Introduce communication through REST API between the clients and the operator coordinators
godfreyhe commented on a change in pull request #12037:
URL: https://github.com/apache/flink/pull/12037#discussion_r422751031
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorSchedulerTest.java
##
@@ -163,6 +163,36 @@ public void taskTaskManagerFailuresAreReportedBack()
throws Exception {
assertThat(result, futureFailedWith(TestException.class));
}
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testDeliveringClientRequestToResponser() throws Exception {
+ final OperatorCoordinator.Provider provider = new
TestingCoordinationResponser.Provider(testOperatorId);
+ final DefaultScheduler scheduler = createScheduler(provider);
+
+ final String payload = "testing payload";
+ final TestingCoordinationResponser.Request request =
+ new TestingCoordinationResponser.Request<>(payload);
+ final TestingCoordinationResponser.Response response =
+ (TestingCoordinationResponser.Response)
+
scheduler.deliverCoordinationRequestToCoordinator(testOperatorId,
request).get();
+
+ assertEquals(payload, response.getPayload());
+ }
+
+ @Test
+ public void testDeliveringClientRequestToNonResponser() throws
Exception {
+ final OperatorCoordinator.Provider provider = new
TestingOperatorCoordinator.Provider(testOperatorId);
+ final DefaultScheduler scheduler = createScheduler(provider);
+
+ final String payload = "testing payload";
+ final TestingCoordinationResponser.Request request =
+ new TestingCoordinationResponser.Request<>(payload);
+ final CompletableFuture future =
+
scheduler.deliverCoordinationRequestToCoordinator(testOperatorId, request);
+
+ assertThat(future,
futureFailedWith(IllegalArgumentException.class));
+ }
+
Review comment:
add a test for scenario: "operator id not found"
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java
##
@@ -933,6 +941,22 @@ public void deliverOperatorEventToCoordinator(
}
}
+ @Override
+ public CompletableFuture
deliverCoordinationRequestToCoordinator(
+ OperatorID operator,
+ CoordinationRequest request) throws FlinkException {
+ OperatorCoordinator coordinator = coordinatorMap.get(operator);
+ if (coordinator instanceof CoordinationResponser) {
+ return CompletableFuture.completedFuture(
+ ((CoordinationResponser)
coordinator).handleCoordinationRequest(request));
+ } else if (coordinator != null) {
+ return FutureUtils.completedExceptionally(
+ new IllegalArgumentException("Coordinator of
operator " + operator + " cannot handle client event"));
Review comment:
I think we should throw `FlinkException`, this situation is the same as
the operation does not exist
##
File path:
flink-clients/src/main/java/org/apache/flink/client/program/rest/RestClusterClient.java
##
@@ -411,6 +418,36 @@ public void close() {
return triggerSavepoint(jobId, savepointDirectory, false);
}
+ @Override
+ public CompletableFuture sendCoordinationRequest(
+ JobID jobId,
+ OperatorID operatorId,
+ CoordinationRequest request) {
+ ClientCoordinationHeaders headers =
ClientCoordinationHeaders.getInstance();
+ ClientCoordinationMessageParameters params = new
ClientCoordinationMessageParameters();
+ params.jobPathParameter.resolve(jobId);
+ params.operatorPathParameter.resolve(operatorId);
+
+ SerializedValue serializedRequest;
+ try {
+ serializedRequest = new SerializedValue<>(request);
+ } catch (IOException e) {
+ return FutureUtils.completedExceptionally(e);
+ }
+
+ ClientCoordinationRequestBody requestBody = new
ClientCoordinationRequestBody(serializedRequest);
+ return sendRequest(headers, params, requestBody).thenApply(
+ responseBody -> {
+ try {
+ return responseBody
+
.getSerializedCoordinationResponse()
+
.deserializeValue(getClass().getClassLoader());
+ } catch (IOException | ClassNotFoundException
e) {
+ throw
[GitHub] [flink] flinkbot edited a comment on pull request #12063: [FLINK-17603][table][core] Prepare Hive partitioned streaming source
flinkbot edited a comment on pull request #12063: URL: https://github.com/apache/flink/pull/12063#issuecomment-626441217 ## CI report: * 11d3939cde0deeac80e4642bb946622e7ee4d850 UNKNOWN * 64ffbab011482e857ba986b2116d9dfb94138259 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12010: [FLINK-17286][connectors / filesystem]Integrate json to file system connector
flinkbot edited a comment on pull request #12010: URL: https://github.com/apache/flink/pull/12010#issuecomment-624746999 ## CI report: * d7445e68ed77a6b1b88fdf0dac8638ff62ac8967 UNKNOWN * 410e972bd87875f3ebc7eb8c3b1f39279397473a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=902) * edacdf8f0062fdae68a1581936a7410c1b35baa2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=926) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17521) Remove `com.ibm.icu` dependency from table-common
[ https://issues.apache.org/jira/browse/FLINK-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104026#comment-17104026 ] Jark Wu commented on FLINK-17521: - I found that `com.ibm.icu` contains some classes with the same name with JDK, e.g. BigDecimal, TimeZone. I think we should get rid of it (at least from the table-common). > Remove `com.ibm.icu` dependency from table-common > - > > Key: FLINK-17521 > URL: https://issues.apache.org/jira/browse/FLINK-17521 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Timo Walther >Priority: Major > > The `com.ibm.icu` dependency has been added recently to the `table-common` > module. > Since this module is used by many connectors and libraries, we should discuss > to remove it again. > Especially, because now there is also another `Row.of()` in the classpath of > the API. Which can be very confusing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
JingsongLi commented on a change in pull request #12004:
URL: https://github.com/apache/flink/pull/12004#discussion_r422753767
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/filesystem/FileSystemOptions.java
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.filesystem;
+
+import org.apache.flink.configuration.ConfigOption;
+
+import java.time.Duration;
+
+import static org.apache.flink.configuration.ConfigOptions.key;
+
+/**
+ * This class holds configuration constants used by filesystem(Including hive)
connector.
+ */
+public class FileSystemOptions {
+
+ public static final ConfigOption STREAMING_SOURCE_ENABLE =
+ key("streaming-source.enable")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription("Enable streaming
source or not.");
+
+ public static final ConfigOption
STREAMING_SOURCE_MONITOR_INTERVAL =
+ key("streaming-source.monitor-interval")
+ .durationType()
+ .defaultValue(Duration.ofMillis(1))
+ .withDescription("The minimum interval
allowed between consecutive partition/file discovery.");
+
+ public static final ConfigOption STREAMING_SOURCE_CONSUME_ORDER
=
+ key("streaming-source.consume-order")
+ .stringType()
+ .defaultValue("create-time")
+ .withDescription("The consume order of
streaming source," +
+ " support create-time
and partition-time." +
+ " create-time compare
partition/file creation time;" +
+ " partition-time
compare time represented by partition name.");
+
+ public static final ConfigOption
STREAMING_SOURCE_CONSUME_START_OFFSET =
+ key("streaming-source.consume-start-offset")
+ .stringType()
+ .defaultValue("1970-00-00")
+ .withDescription("Start offset for
streaming consuming." +
+ " How to parse and
compare offsets depends on your order." +
+ " For create-time and
partition-time, should be a timestamp string.");
+
+ public static final ConfigOption PARTITION_TIME_EXTRACTOR_TYPE =
+ key("partition.time-extractor.type")
+ .stringType()
+ .defaultValue("timestamp")
+ .withDescription("Time extractor to
extract time from partition values." +
+ " Support timestamp and
custom." +
Review comment:
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
JingsongLi commented on a change in pull request #12004:
URL: https://github.com/apache/flink/pull/12004#discussion_r422753438
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/filesystem/FileSystemOptions.java
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.filesystem;
+
+import org.apache.flink.configuration.ConfigOption;
+
+import java.time.Duration;
+
+import static org.apache.flink.configuration.ConfigOptions.key;
+
+/**
+ * This class holds configuration constants used by filesystem(Including hive)
connector.
+ */
+public class FileSystemOptions {
+
+ public static final ConfigOption STREAMING_SOURCE_ENABLE =
+ key("streaming-source.enable")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription("Enable streaming
source or not.");
+
+ public static final ConfigOption
STREAMING_SOURCE_MONITOR_INTERVAL =
+ key("streaming-source.monitor-interval")
+ .durationType()
+ .defaultValue(Duration.ofMillis(1))
+ .withDescription("The minimum interval
allowed between consecutive partition/file discovery.");
+
+ public static final ConfigOption STREAMING_SOURCE_CONSUME_ORDER
=
+ key("streaming-source.consume-order")
+ .stringType()
+ .defaultValue("create-time")
+ .withDescription("The consume order of
streaming source," +
+ " support create-time
and partition-time." +
+ " create-time compare
partition/file creation time;" +
+ " partition-time
compare time represented by partition name.");
+
+ public static final ConfigOption
STREAMING_SOURCE_CONSUME_START_OFFSET =
+ key("streaming-source.consume-start-offset")
+ .stringType()
+ .defaultValue("1970-00-00")
+ .withDescription("Start offset for
streaming consuming." +
+ " How to parse and
compare offsets depends on your order." +
+ " For create-time and
partition-time, should be a timestamp string.");
+
+ public static final ConfigOption PARTITION_TIME_EXTRACTOR_TYPE =
+ key("partition.time-extractor.type")
+ .stringType()
+ .defaultValue("timestamp")
+ .withDescription("Time extractor to
extract time from partition values." +
Review comment:
Yes, will add comments.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
JingsongLi commented on a change in pull request #12004:
URL: https://github.com/apache/flink/pull/12004#discussion_r422753019
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/filesystem/FileSystemOptions.java
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.filesystem;
+
+import org.apache.flink.configuration.ConfigOption;
+
+import java.time.Duration;
+
+import static org.apache.flink.configuration.ConfigOptions.key;
+
+/**
+ * This class holds configuration constants used by filesystem(Including hive)
connector.
+ */
+public class FileSystemOptions {
+
+ public static final ConfigOption STREAMING_SOURCE_ENABLE =
+ key("streaming-source.enable")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription("Enable streaming
source or not.");
+
+ public static final ConfigOption
STREAMING_SOURCE_MONITOR_INTERVAL =
+ key("streaming-source.monitor-interval")
+ .durationType()
+ .defaultValue(Duration.ofMillis(1))
+ .withDescription("The minimum interval
allowed between consecutive partition/file discovery.");
Review comment:
`Time interval for consecutively monitoring partition/file.`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
JingsongLi commented on a change in pull request #12004:
URL: https://github.com/apache/flink/pull/12004#discussion_r422752654
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/TimestampedHiveInputSplit.java
##
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.hive.read;
+
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInfoFactory;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.api.java.typeutils.runtime.DataInputViewStream;
+import org.apache.flink.api.java.typeutils.runtime.DataOutputViewStream;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.streaming.api.functions.source.TimestampedInputSplit;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.lang.reflect.Type;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * A {@link HiveTableInputSplit} with {@link TimestampedInputSplit}.
+ * Kryo serializer can not deal with hadoop split, need specific type
information factory.
+ */
+@TypeInfo(TimestampedHiveInputSplit.SplitTypeInfoFactory.class)
+public class TimestampedHiveInputSplit extends HiveTableInputSplit implements
TimestampedInputSplit {
+
+ private static final long serialVersionUID = 1L;
+
+ /** The modification time of the file this split belongs to. */
+ private final long modificationTime;
+
+ /**
+* The state of the split. This information is used when
+* restoring from a checkpoint and allows to resume reading the
+* underlying file from the point we left off.
+* */
+ private Serializable splitState;
+
+ public TimestampedHiveInputSplit(
+ long modificationTime,
+ HiveTableInputSplit split) {
+ super(
+ split.getSplitNumber(),
+ split.getHadoopInputSplit(),
+ split.getJobConf(),
+ split.getHiveTablePartition());
+ this.modificationTime = modificationTime;
+ }
+
+ @Override
+ public void setSplitState(Serializable state) {
+ this.splitState = state;
+ }
+
+ @Override
+ public Serializable getSplitState() {
+ return this.splitState;
+ }
+
+ @Override
+ public long getModificationTime() {
+ return modificationTime;
+ }
+
+ @Override
+ public int compareTo(TimestampedInputSplit o) {
+ TimestampedHiveInputSplit split = (TimestampedHiveInputSplit) o;
+ int modTimeComp = Long.compare(this.modificationTime,
split.modificationTime);
+ if (modTimeComp != 0L) {
+ return modTimeComp;
+ }
+
+ int pathComp =
this.hiveTablePartition.getStorageDescriptor().compareTo(
+
split.hiveTablePartition.getStorageDescriptor());
+
+ return pathComp != 0 ? pathComp :
+ this.getSplitNumber() - o.getSplitNumber();
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ if (!super.equals(o)) {
Review comment:
Yes, we can create the case like you said, but in read world, it should
not happen.
This is an automated
[GitHub] [flink] JingsongLi commented on a change in pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
JingsongLi commented on a change in pull request #12004:
URL: https://github.com/apache/flink/pull/12004#discussion_r422752473
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/TimestampedHiveInputSplit.java
##
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connectors.hive.read;
+
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInfoFactory;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.api.java.typeutils.runtime.DataInputViewStream;
+import org.apache.flink.api.java.typeutils.runtime.DataOutputViewStream;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.streaming.api.functions.source.TimestampedInputSplit;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.lang.reflect.Type;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * A {@link HiveTableInputSplit} with {@link TimestampedInputSplit}.
+ * Kryo serializer can not deal with hadoop split, need specific type
information factory.
+ */
+@TypeInfo(TimestampedHiveInputSplit.SplitTypeInfoFactory.class)
+public class TimestampedHiveInputSplit extends HiveTableInputSplit implements
TimestampedInputSplit {
+
+ private static final long serialVersionUID = 1L;
+
+ /** The modification time of the file this split belongs to. */
+ private final long modificationTime;
+
+ /**
+* The state of the split. This information is used when
+* restoring from a checkpoint and allows to resume reading the
+* underlying file from the point we left off.
+* */
+ private Serializable splitState;
+
+ public TimestampedHiveInputSplit(
+ long modificationTime,
+ HiveTableInputSplit split) {
+ super(
+ split.getSplitNumber(),
+ split.getHadoopInputSplit(),
+ split.getJobConf(),
+ split.getHiveTablePartition());
+ this.modificationTime = modificationTime;
+ }
+
+ @Override
+ public void setSplitState(Serializable state) {
+ this.splitState = state;
+ }
+
+ @Override
+ public Serializable getSplitState() {
+ return this.splitState;
+ }
+
+ @Override
+ public long getModificationTime() {
+ return modificationTime;
+ }
+
+ @Override
+ public int compareTo(TimestampedInputSplit o) {
+ TimestampedHiveInputSplit split = (TimestampedHiveInputSplit) o;
+ int modTimeComp = Long.compare(this.modificationTime,
split.modificationTime);
+ if (modTimeComp != 0L) {
+ return modTimeComp;
+ }
+
+ int pathComp =
this.hiveTablePartition.getStorageDescriptor().compareTo(
Review comment:
It is a short cut for comparing. If the sd is same, `SplitNumber` must
be different.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
JingsongLi commented on a change in pull request #12004:
URL: https://github.com/apache/flink/pull/12004#discussion_r422751974
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/filesystem/FileSystemOptions.java
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.filesystem;
+
+import org.apache.flink.configuration.ConfigOption;
+
+import java.time.Duration;
+
+import static org.apache.flink.configuration.ConfigOptions.key;
+
+/**
+ * This class holds configuration constants used by filesystem(Including hive)
connector.
+ */
+public class FileSystemOptions {
+
+ public static final ConfigOption STREAMING_SOURCE_ENABLE =
Review comment:
No, but streaming execution mode can run both streaming source and batch
source.
Batch execution mode is a subset of streaming execution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (FLINK-17604) Implement format factory for CSV serialization and deseriazation schema of RowData type
[ https://issues.apache.org/jira/browse/FLINK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-17604: --- Assignee: Danny Chen > Implement format factory for CSV serialization and deseriazation schema of > RowData type > --- > > Key: FLINK-17604 > URL: https://issues.apache.org/jira/browse/FLINK-17604 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.11.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12062: [FLINK-17587][filesystem] Filesystem streaming sink support commit success file
flinkbot edited a comment on pull request #12062: URL: https://github.com/apache/flink/pull/12062#issuecomment-626437386 ## CI report: * 38d4e669261a6fde7595223bf804b45e838476b2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=927) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17146) Support conversion between PyFlink Table and Pandas DataFrame
[ https://issues.apache.org/jira/browse/FLINK-17146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated FLINK-17146: Priority: Blocker (was: Major) > Support conversion between PyFlink Table and Pandas DataFrame > - > > Key: FLINK-17146 > URL: https://issues.apache.org/jira/browse/FLINK-17146 > Project: Flink > Issue Type: New Feature > Components: API / Python >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Blocker > Fix For: 1.11.0 > > > Pandas dataframe is the de-facto standard to work with tabular data in Python > community. PyFlink table is Flink’s representation of the tabular data in > Python language. It would be nice to provide the ability to convert between > the PyFlink table and Pandas dataframe in PyFlink Table API which has the > following benefits: > * It provides users the ability to switch between PyFlink and Pandas > seamlessly when processing data in Python language. Users could process data > using one execution engine and switch to another seamlessly. For example, it > may happen that users have already got a Pandas dataframe at hand and want to > perform some expensive transformation of it. Then they could convert it to a > PyFlink table and leverage the power of Flink engine. Users could also > convert a PyFlink table to Pandas dataframe and perform transformation of it > with the rich functionalities provided by the Pandas ecosystem. > * No intermediate connectors are needed when converting between them. > More details could be found in > [FLIP-120|https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12063: [FLINK-17603][table][core] Prepare Hive partitioned streaming source
flinkbot commented on pull request #12063: URL: https://github.com/apache/flink/pull/12063#issuecomment-626441217 ## CI report: * 11d3939cde0deeac80e4642bb946622e7ee4d850 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12059: [FLINK-17602][docs] Corrected method name in dev/stream/broadcast_sta…
flinkbot edited a comment on pull request #12059: URL: https://github.com/apache/flink/pull/12059#issuecomment-626403738 ## CI report: * f4cf0b903582264f5f5b53e23b3c86d19e213450 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=920) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11297: [FLINK-16364] [table] Deprecate the methods in TableEnvironment proposed by FLIP-84
flinkbot edited a comment on pull request #11297: URL: https://github.com/apache/flink/pull/11297#issuecomment-593931358 ## CI report: * bbf0bc3e167ac3aa30f1044a352f4bdac4a416a2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=923) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12010: [FLINK-17286][connectors / filesystem]Integrate json to file system connector
flinkbot edited a comment on pull request #12010: URL: https://github.com/apache/flink/pull/12010#issuecomment-624746999 ## CI report: * d7445e68ed77a6b1b88fdf0dac8638ff62ac8967 UNKNOWN * 410e972bd87875f3ebc7eb8c3b1f39279397473a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=902) * edacdf8f0062fdae68a1581936a7410c1b35baa2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-17604) Implement format factory for CSV serialization and deseriazation schema of RowData type
Danny Chen created FLINK-17604: -- Summary: Implement format factory for CSV serialization and deseriazation schema of RowData type Key: FLINK-17604 URL: https://issues.apache.org/jira/browse/FLINK-17604 Project: Flink Issue Type: Sub-task Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Affects Versions: 1.11.0 Reporter: Danny Chen Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on pull request #11687: [FLINK-16536][network][checkpointing] Implement InputChannel state recovery for unaligned checkpoint
zhijiangW commented on pull request #11687: URL: https://github.com/apache/flink/pull/11687#issuecomment-626440767 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12063: [FLINK-17603][table][core] Prepare Hive partitioned streaming source
flinkbot commented on pull request #12063: URL: https://github.com/apache/flink/pull/12063#issuecomment-626439751 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 11d3939cde0deeac80e4642bb946622e7ee4d850 (Mon May 11 02:38:16 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17603) Prepare Hive partitioned streaming source
[ https://issues.apache.org/jira/browse/FLINK-17603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17603: --- Labels: pull-request-available (was: ) > Prepare Hive partitioned streaming source > - > > Key: FLINK-17603 > URL: https://issues.apache.org/jira/browse/FLINK-17603 > Project: Flink > Issue Type: Sub-task >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > * Refactor ContinuousFileReaderOperator to read generic split > * HiveTableInputFormat implements CheckpointableInputFormat -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi opened a new pull request #12063: [FLINK-17603][table][core] Prepare Hive partitioned streaming source
JingsongLi opened a new pull request #12063: URL: https://github.com/apache/flink/pull/12063 ## What is the purpose of the change Prepare Hive partitioned streaming source, refactor codes. ## Brief change log - Refactor ContinuousFileReaderOperator to read generic split - HiveTableInputFormat implements CheckpointableInputFormat ## Verifying this change This change is already covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency):no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive):no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented?JavaDocs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #12004: [FLINK-17434][core][hive] Hive partitioned source support streaming read
JingsongLi commented on a change in pull request #12004:
URL: https://github.com/apache/flink/pull/12004#discussion_r422748256
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/HiveTableInputFormat.java
##
@@ -248,7 +263,11 @@ public RowData nextRecord(RowData reuse) throws
IOException {
}
@Override
- public void close() throws IOException {
+ public void close() {
Review comment:
I am wrong, should in close.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dianfu commented on a change in pull request #12061: [FLINK-17454][python]python gateway callback server port should not b…
dianfu commented on a change in pull request #12061:
URL: https://github.com/apache/flink/pull/12061#discussion_r422746320
##
File path:
flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java
##
@@ -300,6 +294,48 @@ static GatewayServer startGatewayServer() throws
ExecutionException, Interrupted
return gatewayServerFuture.get();
}
+ /**
+* Reset a daemon thread to the callback client thread pool so that the
callback server can be terminated when gate
+* way server is shutting down. We need to shut down the none-daemon
thread firstly, then set a new thread created
+* in a daemon thread to the ExecutorService.
+*
+* @param gatewayServer the gateway which creates the callback server.
+* */
+ private static void resetCallbackClientExecutorService(GatewayServer
gatewayServer) throws NoSuchFieldException,
+ IllegalAccessException, NoSuchMethodException,
InvocationTargetException {
+ CallbackClient callbackClient = (CallbackClient)
gatewayServer.getCallbackClient();
+ // The Java API of py4j does not provide approach to set
"daemonize_connections" parameter.
+ // Use reflect to daemonize the connection thread.
+ Field executor =
CallbackClient.class.getDeclaredField("executor");
+ executor.setAccessible(true);
+ ((ScheduledExecutorService)
executor.get(callbackClient)).shutdown();
+ executor.set(callbackClient,
Executors.newScheduledThreadPool(1, Thread::new));
+ Method setupCleaner =
CallbackClient.class.getDeclaredMethod("setupCleaner");
+ setupCleaner.setAccessible(true);
+ setupCleaner.invoke(callbackClient);
+ }
+
+ /**
+* Reset the callback client of gatewayServer with the given
callbackListeningAddress and callbackListeningPort
+* after the callback server started.
+*
+* @param callbackListeningAddress the listening address of the
callback server.
+* @param callbackListeningPort the listening port of the callback
server.
+* */
+ public static void resetCallbackClient(String callbackListeningAddress,
int callbackListeningPort) throws
+ UnknownHostException, InvocationTargetException,
NoSuchMethodException, IllegalAccessException,
+ NoSuchFieldException {
+
+ gatewayServer = getGatewayServer();
+ if (gatewayServer == null){
Review comment:
It seems that it is never `null`.
##
File path: flink-python/pyflink/java_gateway.py
##
@@ -49,15 +49,19 @@ def get_gateway():
# if Java Gateway is already running
if 'PYFLINK_GATEWAY_PORT' in os.environ:
gateway_port = int(os.environ['PYFLINK_GATEWAY_PORT'])
-callback_port = int(os.environ['PYFLINK_CALLBACK_PORT'])
gateway_param = GatewayParameters(port=gateway_port,
auto_convert=True)
_gateway = JavaGateway(
gateway_parameters=gateway_param,
callback_server_parameters=CallbackServerParameters(
-port=callback_port, daemonize=True,
daemonize_connections=True))
+port=0, daemonize=True, daemonize_connections=True))
else:
_gateway = launch_gateway()
+callback_server = _gateway.get_callback_server()
+listening_callback_address =
callback_server.get_listening_address()
Review comment:
rename to `callback_server_listening_address` and
`callback_server_listening_port `
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12062: [FLINK-17587][filesystem] Filesystem streaming sink support commit success file
flinkbot commented on pull request #12062: URL: https://github.com/apache/flink/pull/12062#issuecomment-626437386 ## CI report: * 38d4e669261a6fde7595223bf804b45e838476b2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12049: [FLINK-17252] [table] supports SELECT statement in TableEnvironment#executeSql and introduce Table#execute api
flinkbot edited a comment on pull request #12049: URL: https://github.com/apache/flink/pull/12049#issuecomment-626106371 ## CI report: * 50dfd19d62da7a39f2004934703344b7f3d95573 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=921) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12061: [FLINK-17454][python]python gateway callback server port should not b…
flinkbot edited a comment on pull request #12061: URL: https://github.com/apache/flink/pull/12061#issuecomment-626432764 ## CI report: * d07c04b5ac978764112af3bf1d35285195e3440d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=924) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-17603) Prepare Hive partitioned streaming source
Jingsong Lee created FLINK-17603: Summary: Prepare Hive partitioned streaming source Key: FLINK-17603 URL: https://issues.apache.org/jira/browse/FLINK-17603 Project: Flink Issue Type: Sub-task Reporter: Jingsong Lee Assignee: Jingsong Lee Fix For: 1.11.0 * Refactor ContinuousFileReaderOperator to read generic split * HiveTableInputFormat implements CheckpointableInputFormat -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #11297: [FLINK-16364] [table] Deprecate the methods in TableEnvironment proposed by FLIP-84
flinkbot edited a comment on pull request #11297: URL: https://github.com/apache/flink/pull/11297#issuecomment-593931358 ## CI report: * a62c733033b7de8e7d61271da25a465ec544e78c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=917) * bbf0bc3e167ac3aa30f1044a352f4bdac4a416a2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=923) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-17601) TableEnvironmentITCase.testStatementSet fails on travis
[
https://issues.apache.org/jira/browse/FLINK-17601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-17601:
---
Assignee: godfrey he
> TableEnvironmentITCase.testStatementSet fails on travis
> ---
>
> Key: FLINK-17601
> URL: https://issues.apache.org/jira/browse/FLINK-17601
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: Dawid Wysakowicz
>Assignee: godfrey he
>Priority: Blocker
> Fix For: 1.11.0
>
>
> {code}
> [ERROR] Failures:
> [ERROR] TableEnvironmentITCase.testStatementSet:446 expected:<...
> fields=[first])
> +- [TableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- ]TableSourceScan(tabl...> but was:<... fields=[first])
> +- [LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- Legacy]TableSourceScan(tabl...>
> [ERROR] TableEnvironmentITCase.testStatementSet:446 expected:<...
> fields=[first])
> +- [TableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- ]TableSourceScan(tabl...> but was:<... fields=[first])
> +- [LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- Legacy]TableSourceScan(tabl...>
> [ERROR] TableEnvironmentITCase.testStatementSet:446 expected:<...
> fields=[first])
> +- [TableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- ]TableSourceScan(tabl...> but was:<... fields=[first])
> +- [LegacyTableSourceScan(table=[[default_catalog, default_database, MyTable,
> source: [CsvTableSource(read fields: first)]]], fields=[first])
> Sink(name=[`default_catalog`.`default_database`.`MySink2`], fields=[last])
> +- Legacy]TableSourceScan(tabl...>
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=912=logs=e25d5e7e-2a9c-5589-4940-0b638d75a414=f83cd372-208c-5ec4-12a8-337462457129
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17565) Bump fabric8 version from 4.5.2 to 4.9.1
[ https://issues.apache.org/jira/browse/FLINK-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104007#comment-17104007 ] Canbin Zheng commented on FLINK-17565: -- Hi [~tison]! At the moment this ticket first aims to solve the issue of [FLINK-17566|https://issues.apache.org/jira/browse/FLINK-17566]. For advanced features such as LeaderElection support, maybe we need more investigation and verification before coming up with concrete plans. Besides, for the HA implementation in native K8s setups, I think maybe an HA service based on FileSystem is a good start point. > Bump fabric8 version from 4.5.2 to 4.9.1 > > > Key: FLINK-17565 > URL: https://issues.apache.org/jira/browse/FLINK-17565 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Reporter: Canbin Zheng >Priority: Major > > Currently, we are using a version of 4.5.2, it's better that we upgrade it to > 4.9.1 for features like K8s 1.17 support, LeaderElection support, and etc. > For more details, please refer to [fabric8 > releases|https://github.com/fabric8io/kubernetes-client/releases]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17452) Support creating Hive tables with constraints
[ https://issues.apache.org/jira/browse/FLINK-17452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-17452: - Fix Version/s: 1.11.0 > Support creating Hive tables with constraints > - > > Key: FLINK-17452 > URL: https://issues.apache.org/jira/browse/FLINK-17452 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-17452) Support creating Hive tables with constraints
[ https://issues.apache.org/jira/browse/FLINK-17452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-17452: Assignee: Rui Li > Support creating Hive tables with constraints > - > > Key: FLINK-17452 > URL: https://issues.apache.org/jira/browse/FLINK-17452 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-17452) Support creating Hive tables with constraints
[ https://issues.apache.org/jira/browse/FLINK-17452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-17452. Resolution: Fixed master: 25cbce12d7201642b6031e38583489cd803fb1a5 > Support creating Hive tables with constraints > - > > Key: FLINK-17452 > URL: https://issues.apache.org/jira/browse/FLINK-17452 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Rui Li >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on pull request #12017: [FLINK-17452][hive] Support creating Hive tables with constraints
JingsongLi commented on pull request #12017: URL: https://github.com/apache/flink/pull/12017#issuecomment-626434551 Thanks for updating. Merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12062: [FLINK-17587][filesystem] Filesystem streaming sink support commit success file
flinkbot commented on pull request #12062: URL: https://github.com/apache/flink/pull/12062#issuecomment-626434311 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 38d4e669261a6fde7595223bf804b45e838476b2 (Mon May 11 02:12:22 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-17587) Filesystem streaming sink support partition commit (success file)
[ https://issues.apache.org/jira/browse/FLINK-17587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-17587: Assignee: Jingsong Lee > Filesystem streaming sink support partition commit (success file) > - > > Key: FLINK-17587 > URL: https://issues.apache.org/jira/browse/FLINK-17587 > Project: Flink > Issue Type: Sub-task >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17587) Filesystem streaming sink support partition commit (success file)
[ https://issues.apache.org/jira/browse/FLINK-17587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17587: --- Labels: pull-request-available (was: ) > Filesystem streaming sink support partition commit (success file) > - > > Key: FLINK-17587 > URL: https://issues.apache.org/jira/browse/FLINK-17587 > Project: Flink > Issue Type: Sub-task >Reporter: Jingsong Lee >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi opened a new pull request #12062: [FLINK-17587][filesystem] Filesystem streaming sink support commit success file
JingsongLi opened a new pull request #12062: URL: https://github.com/apache/flink/pull/12062 ## What is the purpose of the change Committing a partition is to notify the downstream application that the partition has finished writing, the partition is ready to be read. Add “.succes” file to directory (success file name is configurable too) ## Brief change log - This PR is based on https://github.com/apache/flink/pull/12053 - StreamingFileWriter(parallelism) -> StreamingFileCommitter(Single task to commit partition) ## Verifying this change `StreamFileSystemTestCsvITCase` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? JavaDocs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12060: [hotfix] [table-planner-blink] correct the table scan node name in the explain result of TableEnvironmentITCase#testStatementSet
flinkbot edited a comment on pull request #12060: URL: https://github.com/apache/flink/pull/12060#issuecomment-626425753 ## CI report: * ad83387d0a550034ecfe9daa5a8f7b644e195954 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=922) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
