[GitHub] [flink] flinkbot edited a comment on pull request #13158: [hotfix] [docs] Fix typo in checkpoint monitoring doc
flinkbot edited a comment on pull request #13158: URL: https://github.com/apache/flink/pull/13158#issuecomment-674472407 ## CI report: * 6117c8b294a5fb0df0ec7fa5e95377890a650754 UNKNOWN * e29749fef5d6062e2a9290890d5863ef73265347 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18967) Upgrade ORC to 1.5.10 version
[ https://issues.apache.org/jira/browse/FLINK-18967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated FLINK-18967: -- Attachment: FLINK-18967.patch > Upgrade ORC to 1.5.10 version > - > > Key: FLINK-18967 > URL: https://issues.apache.org/jira/browse/FLINK-18967 > Project: Flink > Issue Type: Task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.11.1 >Reporter: Igor Dvorzhak >Priority: Major > Attachments: FLINK-18967.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13159: [hotfix] [docs] Fix typo for multiple doc files
flinkbot edited a comment on pull request #13159: URL: https://github.com/apache/flink/pull/13159#issuecomment-674476706 ## CI report: * 1f2668e77f7c46e7e7b5f70a2944f37e27f4d038 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5558) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13159: [hotfix] [docs] Fix typo for multiple doc files
flinkbot commented on pull request #13159: URL: https://github.com/apache/flink/pull/13159#issuecomment-674476706 ## CI report: * 1f2668e77f7c46e7e7b5f70a2944f37e27f4d038 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18967) Upgrade ORC to 1.5.10 version
Igor Dvorzhak created FLINK-18967: - Summary: Upgrade ORC to 1.5.10 version Key: FLINK-18967 URL: https://issues.apache.org/jira/browse/FLINK-18967 Project: Flink Issue Type: Task Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Affects Versions: 1.11.1 Reporter: Igor Dvorzhak -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12532) Upgrade Avro to version 1.9.2
[ https://issues.apache.org/jira/browse/FLINK-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated FLINK-12532: -- Attachment: FLINK-18892.patch > Upgrade Avro to version 1.9.2 > - > > Key: FLINK-12532 > URL: https://issues.apache.org/jira/browse/FLINK-12532 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Ismaël Mejía >Priority: Minor > Attachments: FLINK-12532.patch > > > Avro 1.9.2 was released with many nice features including reduced size (1MB > less), and removed dependencies, no paranmer, no shaded guava, security > updates, so probably a worth upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12532) Upgrade Avro to version 1.9.2
[ https://issues.apache.org/jira/browse/FLINK-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated FLINK-12532: -- Attachment: FLINK-12532.patch > Upgrade Avro to version 1.9.2 > - > > Key: FLINK-12532 > URL: https://issues.apache.org/jira/browse/FLINK-12532 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Ismaël Mejía >Priority: Minor > Attachments: FLINK-12532.patch > > > Avro 1.9.2 was released with many nice features including reduced size (1MB > less), and removed dependencies, no paranmer, no shaded guava, security > updates, so probably a worth upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12532) Upgrade Avro to version 1.9.2
[ https://issues.apache.org/jira/browse/FLINK-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated FLINK-12532: -- Attachment: (was: FLINK-18892.patch) > Upgrade Avro to version 1.9.2 > - > > Key: FLINK-12532 > URL: https://issues.apache.org/jira/browse/FLINK-12532 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Ismaël Mejía >Priority: Minor > Attachments: FLINK-12532.patch > > > Avro 1.9.2 was released with many nice features including reduced size (1MB > less), and removed dependencies, no paranmer, no shaded guava, security > updates, so probably a worth upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13159: [hotfix] [docs] Fix typo for multiple doc files
flinkbot commented on pull request #13159: URL: https://github.com/apache/flink/pull/13159#issuecomment-674475693 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1f2668e77f7c46e7e7b5f70a2944f37e27f4d038 (Sun Aug 16 04:01:58 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] brandonJY opened a new pull request #13159: [hotfix] [docs] Fix typo for multiple doc files
brandonJY opened a new pull request #13159: URL: https://github.com/apache/flink/pull/13159 Fix typo for multiple doc files This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13158: [hotfix] [docs] Fix typo in checkpoint monitoring doc
flinkbot edited a comment on pull request #13158: URL: https://github.com/apache/flink/pull/13158#issuecomment-674472407 ## CI report: * 6117c8b294a5fb0df0ec7fa5e95377890a650754 UNKNOWN * e29749fef5d6062e2a9290890d5863ef73265347 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18959) Fail to archiveExecutionGraph because job is not finished when dispatcher close
[
https://issues.apache.org/jira/browse/FLINK-18959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17178414#comment-17178414
]
Liu commented on FLINK-18959:
-
Thank you, [~trohrmann]. I have the same confusion with you. In previous
implementation, job reaches globally terminal state first and then
archiveExecutionGraph is done. Finally, the dispatcher is shut down.
> Fail to archiveExecutionGraph because job is not finished when dispatcher
> close
> ---
>
> Key: FLINK-18959
> URL: https://issues.apache.org/jira/browse/FLINK-18959
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.10.0, 1.12.0, 1.11.1
>Reporter: Liu
>Priority: Major
> Fix For: 1.12.0, 1.11.2, 1.10.3
>
> Attachments: flink-debug-log
>
>
> When job is cancelled, we expect to see it in flink's history server. But I
> can not see my job after it is cancelled.
> After digging into the problem, I find that the function
> archiveExecutionGraph is not executed. Below is the brief log:
> {panel:title=log}
> 2020-08-14 15:10:06,406 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph
> [flink-akka.actor.default-dispatcher- 15] - Job EtlAndWindow
> (6f784d4cc5bae88a332d254b21660372) switched from state RUNNING to CANCELLING.
> 2020-08-14 15:10:06,415 DEBUG
> org.apache.flink.runtime.dispatcher.MiniDispatcher
> [flink-akka.actor.default-dispatcher-3] - Shutting down per-job cluster
> because the job was canceled.
> 2020-08-14 15:10:06,629 INFO
> org.apache.flink.runtime.dispatcher.MiniDispatcher
> [flink-akka.actor.default-dispatcher-3] - Stopping dispatcher
> akka.tcp://flink@bjfk-c9865.yz02:38663/user/dispatcher.
> 2020-08-14 15:10:06,629 INFO
> org.apache.flink.runtime.dispatcher.MiniDispatcher
> [flink-akka.actor.default-dispatcher-3] - Stopping all currently running jobs
> of dispatcher akka.tcp://flink@bjfk-c9865.yz02:38663/user/dispatcher.
> 2020-08-14 15:10:06,631 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [flink-akka.actor.default-dispatcher-29] - Stopping the JobMaster for job
> EtlAndWindow(6f784d4cc5bae88a332d254b21660372).
> 2020-08-14 15:10:06,632 DEBUG org.apache.flink.runtime.jobmaster.JobMaster
> [flink-akka.actor.default-dispatcher-29] - Disconnect TaskExecutor
> container_e144_1590060720089_2161_01_06 because: Stopping JobMaster for
> job EtlAndWindow(6f784d4cc5bae88a332d254b21660372).
> 2020-08-14 15:10:06,646 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph
> [flink-akka.actor.default-dispatcher-29] - Job EtlAndWindow
> (6f784d4cc5bae88a332d254b21660372) switched from state CANCELLING to CANCELED.
> 2020-08-14 15:10:06,664 DEBUG
> org.apache.flink.runtime.dispatcher.MiniDispatcher
> [flink-akka.actor.default-dispatcher-4] - There is a newer JobManagerRunner
> for the job 6f784d4cc5bae88a332d254b21660372.
> {panel}
> From the log, we can see that job is not finished when dispatcher close. The
> process is as following:
> * Receive cancel command and send it to all tasks async.
> * In MiniDispatcher, begin to shutting down per-job cluster.
> * Stopping dispatcher and remove job.
> * Job is cancelled and callback is executed in method startJobManagerRunner.
> * Because job is removed before, so currentJobManagerRunner is null which
> not equals to the original jobManagerRunner. In this case,
> archivedExecutionGraph will not be uploaded.
> In normal cases, I find that job is cancelled first and then dispatcher is
> stopped so that archivedExecutionGraph will succeed. But I think that the
> order is not constrained and it is hard to know which comes first.
> Above is what I suspected. If so, then we should fix it.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13158: [hotfix] [docs] Fix typo in checkpoint monitoring doc
flinkbot edited a comment on pull request #13158: URL: https://github.com/apache/flink/pull/13158#issuecomment-674472407 ## CI report: * 6117c8b294a5fb0df0ec7fa5e95377890a650754 UNKNOWN * e29749fef5d6062e2a9290890d5863ef73265347 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13158: [hotfix] [docs] Fix typo in checkpoint monitoring doc
flinkbot commented on pull request #13158: URL: https://github.com/apache/flink/pull/13158#issuecomment-674472407 ## CI report: * 6117c8b294a5fb0df0ec7fa5e95377890a650754 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13158: [hotfix] [docs] Fix typo in checkpoint monitoring doc
flinkbot commented on pull request #13158: URL: https://github.com/apache/flink/pull/13158#issuecomment-674470723 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6117c8b294a5fb0df0ec7fa5e95377890a650754 (Sun Aug 16 02:53:21 UTC 2020) **Warnings:** * Documentation files were touched, but no `.zh.md` files: Update Chinese documentation or file Jira ticket. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] brandonJY opened a new pull request #13158: [hotfix] [docs] Fix typo in checkpoint monitoring doc
brandonJY opened a new pull request #13158: URL: https://github.com/apache/flink/pull/13158 fix typo in checkpoint_monitoring doc This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13157: [FLINK-18900][hive] HiveCatalog should error out when listing partitions with an invalid spec
flinkbot edited a comment on pull request #13157: URL: https://github.com/apache/flink/pull/13157#issuecomment-674410767 ## CI report: * b64a33a3e57e4af2522aeaf274c4711ddb2b00ef Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5554) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13156: [FLINK-18949][python] Support Streaming File Sink for Python DataStream API
flinkbot edited a comment on pull request #13156: URL: https://github.com/apache/flink/pull/13156#issuecomment-674396240 ## CI report: * 2ee1d96a5033c05e21ddd43004784e63a3fd8c61 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5553) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13157: [FLINK-18900][hive] HiveCatalog should error out when listing partitions with an invalid spec
flinkbot edited a comment on pull request #13157: URL: https://github.com/apache/flink/pull/13157#issuecomment-674410767 ## CI report: * b64a33a3e57e4af2522aeaf274c4711ddb2b00ef Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5554) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13157: [FLINK-18900][hive] HiveCatalog should error out when listing partitions with an invalid spec
flinkbot commented on pull request #13157: URL: https://github.com/apache/flink/pull/13157#issuecomment-674410767 ## CI report: * b64a33a3e57e4af2522aeaf274c4711ddb2b00ef UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13157: [FLINK-18900][hive] HiveCatalog should error out when listing partitions with an invalid spec
flinkbot commented on pull request #13157: URL: https://github.com/apache/flink/pull/13157#issuecomment-674409505 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b64a33a3e57e4af2522aeaf274c4711ddb2b00ef (Sat Aug 15 15:09:50 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18900) HiveCatalog should error out when listing partitions with an invalid spec
[
https://issues.apache.org/jira/browse/FLINK-18900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-18900:
---
Labels: pull-request-available (was: )
> HiveCatalog should error out when listing partitions with an invalid spec
> -
>
> Key: FLINK-18900
> URL: https://issues.apache.org/jira/browse/FLINK-18900
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.11.1
>Reporter: Rui Li
>Assignee: Nicholas Jiang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
>
> Take the following case as an example:
> {code}
> create table tbl (x int) partitioned by (p int);
> alter table tbl add partition (p=1);
> {code}
> If we list partitions with partition spec {{foo=1}}, HiveCatalog returns
> partition {{p=1}}, which is wrong.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] SteNicholas opened a new pull request #13157: [FLINK-18900][hive] HiveCatalog should error out when listing partitions with an invalid spec
SteNicholas opened a new pull request #13157: URL: https://github.com/apache/flink/pull/13157 ## What is the purpose of the change *Currently `HiveCatalog` could return all partitions of table for `SHOW PARTITIONS` with an invalid spec, which is wrong. `HiveCatalog` should error out when listing partitions with an invalid spec.* ## Brief change log - *`HiveCatalog` throws `PartitionNotExistException ` after calling `HiveReflectionUtils.getPvals` with an invalid spec.* ## Verifying this change - *Method `testShowPartitions` of `HiveDialectITCase` add case for executing `SHOW PARTITIONS` with an invalid spec.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] hequn8128 commented on a change in pull request #13155: [FLINK-18947][python] Support partitionCustom() operation for Python …
hequn8128 commented on a change in pull request #13155:
URL: https://github.com/apache/flink/pull/13155#discussion_r470989005
##
File path: flink-python/pyflink/datastream/data_stream.py
##
@@ -434,6 +434,60 @@ def broadcast(self) -> 'DataStream':
"""
return DataStream(self._j_data_stream.broadcast())
+def partition_custom(self, partitioner: Union[Callable, Partitioner],
+ key_selector: Union[Callable, KeySelector]) ->
'DataStream':
+"""
+Partitions a DataStream on the key returned by the selector, using a
custom partitioner.
+This method takes the key selector to get the key to partition on, and
a partitioner that
+accepts the key type.
+
+Note that this method works only on single field keys, i.e. the
selector cannet return
+tuples of fields.
+
+:param partitioner: The partitioner to assign partitions to keys.
+:param key_selector: The KeySelector with which the DataStream is
partitioned.
+:return: The partitioned DataStream.
+"""
+if callable(key_selector):
+key_selector = KeySelectorFunctionWrapper(key_selector)
+if not isinstance(key_selector, (KeySelector,
KeySelectorFunctionWrapper)):
+raise TypeError("Parameter key_selector should be a type of
KeySelector.")
+
+if callable(partitioner):
+partitioner = PartitionerFunctionWrapper(partitioner)
+if not isinstance(partitioner, (Partitioner,
PartitionerFunctionWrapper)):
+raise TypeError("Parameter partitioner should be a type of
Partitioner.")
+
+gateway = get_gateway()
+data_stream_num_partitions_env_key = gateway.jvm\
+.org.apache.flink.datastream.runtime.operators.python\
+
.DataStreamPythonPartitionCustomFunctionOperator.DATA_STREAM_NUM_PARTITIONS
+
+def partition_custom_map(value):
+num_partitions =
int(os.environ[data_stream_num_partitions_env_key])
Review comment:
Can we find ways to init `num_partitions` only once?
##
File path:
flink-python/src/main/java/org/apache/flink/streaming/api/operators/python/AbstractPythonFunctionOperator.java
##
@@ -344,7 +345,7 @@ protected PythonEnvironmentManager
createPythonEnvironmentManager() throws IOExc
return new ProcessPythonEnvironmentManager(
dependencyInfo,
getContainingTask().getEnvironment().getTaskManagerInfo().getTmpDirectories(),
- System.getenv());
+ new HashMap<>(System.getenv()));
Review comment:
Why need this change?
##
File path: flink-python/pyflink/datastream/tests/test_data_stream.py
##
@@ -304,6 +305,28 @@ def test_shuffle(self):
pre_ship_strategy = shuffle_node['predecessors'][0]['ship_strategy']
self.assertEqual(pre_ship_strategy, 'SHUFFLE')
+def test_partition_custom(self):
+ds = self.env.from_collection([('a', 0), ('b', 0), ('c', 1), ('d', 1),
('e', 2)],
Review comment:
add a number which greater than `expected_num_partitions`
##
File path: flink-python/pyflink/datastream/functions.py
##
@@ -324,3 +364,16 @@ def __init__(self, sink_func: Union[str, JavaObject]):
:param sink_func: The java SinkFunction object or the full name of the
SinkFunction class.
"""
super(SinkFunction, self).__init__(sink_func)
+
+
+class PartitionCustomMapFunction(MapFunction):
Review comment:
Add this function here would confuse users since this is not a public
Function. I think we can find other ways to judge whether this is a
`PartitionCustomMapFunction`. For example, we can add a local `MapFunction`
returning a special `__expr__`(toString).
##
File path:
flink-python/src/main/java/org/apache/flink/python/env/beam/ProcessPythonEnvironmentManager.java
##
@@ -240,6 +240,10 @@ public String createRetrievalToken() throws IOException {
return env;
}
+ public void appendEnvironmentVariable(String key, String value) {
Review comment:
Rename to `setEnvironmentVariable`. This method would replace the old
value if key exists.
##
File path: flink-python/pyflink/datastream/functions.py
##
@@ -154,6 +154,22 @@ def filter(self, value):
pass
+class Partitioner(Function):
+"""
+Function to implement a custom partition assignment for keys.
+"""
+
+@abc.abstractmethod
+def partition(self, key, num_partitions: int) -> int:
+"""
+Computes the partition for the given key.
Review comment:
Keep a line break.
##
File path:
flink-python/src/main/java/org/apache/flink/datastream/runtime/functions/python/PartitionCustomPartitioner.java
##
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software
[GitHub] [flink] flinkbot edited a comment on pull request #13154: [FLINK-18704][table] Add DECIMAL type in datagen
flinkbot edited a comment on pull request #13154: URL: https://github.com/apache/flink/pull/13154#issuecomment-674377845 ## CI report: * d8819ee8585dde4ef1da684d8e31f02b964ce6c0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5551) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal edited a comment on pull request #13059: [hotfix][docs] Update the links' tags in the 'Debugging Windows & Event Time' page of 'Debugging & Monitoring'
RocMarshal edited a comment on pull request #13059: URL: https://github.com/apache/flink/pull/13059#issuecomment-67197 Hi, @rmetzger @alpinegizmo I made a hotfix. Please take a look. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13156: [FLINK-18949][python] Support Streaming File Sink for Python DataStream API
flinkbot edited a comment on pull request #13156: URL: https://github.com/apache/flink/pull/13156#issuecomment-674396240 ## CI report: * 2ee1d96a5033c05e21ddd43004784e63a3fd8c61 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5553) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13155: [FLINK-18947][python] Support partitionCustom() operation for Python …
flinkbot edited a comment on pull request #13155: URL: https://github.com/apache/flink/pull/13155#issuecomment-674377857 ## CI report: * 70aeee4d216b47fe0f118abec2758f0a7f6a1260 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5552) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13156: [FLINK-18949][python] Support Streaming File Sink for Python DataStream API
flinkbot commented on pull request #13156: URL: https://github.com/apache/flink/pull/13156#issuecomment-674396240 ## CI report: * 2ee1d96a5033c05e21ddd43004784e63a3fd8c61 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13156: [FLINK-18949][python] Support Streaming File Sink for Python DataStream API
flinkbot commented on pull request #13156: URL: https://github.com/apache/flink/pull/13156#issuecomment-674395551 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 2ee1d96a5033c05e21ddd43004784e63a3fd8c61 (Sat Aug 15 13:18:36 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18949) Support Streaming File Sink for Python DataStream API
[ https://issues.apache.org/jira/browse/FLINK-18949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18949: --- Labels: pull-request-available (was: ) > Support Streaming File Sink for Python DataStream API > - > > Key: FLINK-18949 > URL: https://issues.apache.org/jira/browse/FLINK-18949 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Hequn Cheng >Assignee: Hequn Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] hequn8128 opened a new pull request #13156: [FLINK-18949][python] Support Streaming File Sink for Python DataStream API
hequn8128 opened a new pull request #13156: URL: https://github.com/apache/flink/pull/13156 ## What is the purpose of the change This pull request supports StreamingFileSink on Python DataStream API. Row-encoded Formats is only supported in this PR, i.e., Bulk-encoded Formats are not supported. ## Brief change log - Add StreamingFileSink on Python API. - Add Row-encoded Formats related APIs. ## Verifying this change This change added tests and can be verified as follows: - Added `test_stream_file_sink` integration tests in test_data_stream ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (PythonDocs) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13152: [FLINK-18945][python] Support CoFlatMap for Python DataStream API
flinkbot edited a comment on pull request #13152: URL: https://github.com/apache/flink/pull/13152#issuecomment-674356260 ## CI report: * fb27e2e5684c59966cc505fbdc7969017ee067c9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5548) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13153: [FLINK-18966][python] Support key_by() on ConnectedStreams for Python DataStream API
flinkbot edited a comment on pull request #13153: URL: https://github.com/apache/flink/pull/13153#issuecomment-674367157 ## CI report: * f59963fa5ee56152ca93de32ed73e7fc67eb4250 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5549) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13154: [FLINK-18704][table] Add DECIMAL type in datagen
flinkbot edited a comment on pull request #13154: URL: https://github.com/apache/flink/pull/13154#issuecomment-674377845 ## CI report: * d8819ee8585dde4ef1da684d8e31f02b964ce6c0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5551) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13155: [FLINK-18947][python] Support partitionCustom() operation for Python …
flinkbot edited a comment on pull request #13155: URL: https://github.com/apache/flink/pull/13155#issuecomment-674377857 ## CI report: * 70aeee4d216b47fe0f118abec2758f0a7f6a1260 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5552) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13154: [FLINK-18704][table] Add DECIMAL type in datagen
flinkbot commented on pull request #13154: URL: https://github.com/apache/flink/pull/13154#issuecomment-674377845 ## CI report: * d8819ee8585dde4ef1da684d8e31f02b964ce6c0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13155: [FLINK-18947][python] Support partitionCustom() operation for Python …
flinkbot commented on pull request #13155: URL: https://github.com/apache/flink/pull/13155#issuecomment-674377857 ## CI report: * 70aeee4d216b47fe0f118abec2758f0a7f6a1260 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13155: [FLINK-18947][python] Support partitionCustom() operation for Python …
flinkbot commented on pull request #13155: URL: https://github.com/apache/flink/pull/13155#issuecomment-674375828 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 70aeee4d216b47fe0f118abec2758f0a7f6a1260 (Sat Aug 15 09:45:50 UTC 2020) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18947) Support partitionCustom() operation for Python DataStream API
[ https://issues.apache.org/jira/browse/FLINK-18947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18947: --- Labels: pull-request-available (was: ) > Support partitionCustom() operation for Python DataStream API > - > > Key: FLINK-18947 > URL: https://issues.apache.org/jira/browse/FLINK-18947 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Hequn Cheng >Assignee: shuiqiangchen >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] shuiqiangchen opened a new pull request #13155: [FLINK-18947][python] Support partitionCustom() operation for Python …
shuiqiangchen opened a new pull request #13155: URL: https://github.com/apache/flink/pull/13155 ## What is the purpose of the change Add partition_custom() operation for Python DataStream API, so that users can specify custom partitioning. ## Brief change log - Added partition_custom() for Python DataStream. - Added a new Function named Partitioner. - Make PythonRunner Environment Variables can be appended ## Verifying this change This pull request has been test by test_partition_custom() in test_data_stream.py ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? ( not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13154: [FLINK-18704][table] Add DECIMAL type in datagen
flinkbot commented on pull request #13154: URL: https://github.com/apache/flink/pull/13154#issuecomment-674374845 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit d8819ee8585dde4ef1da684d8e31f02b964ce6c0 (Sat Aug 15 09:37:58 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-18704).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18704) Support DECIMAL types in datagen Table source connector
[
https://issues.apache.org/jira/browse/FLINK-18704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-18704:
---
Labels: pull-request-available starter (was: starter)
> Support DECIMAL types in datagen Table source connector
> ---
>
> Key: FLINK-18704
> URL: https://issues.apache.org/jira/browse/FLINK-18704
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Ecosystem
>Affects Versions: 1.11.0, 1.11.1
>Reporter: Fabian Hueske
>Priority: Major
> Labels: pull-request-available, starter
>
> It would be great if the [{{datagen}} source
> connector|https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/datagen.html]
> would support {{DECIMAL}} types.
> Since data is randomly generated and FLOAT and DOUBLE are supported, we could
> implement this feature by creatig a {{BigDecimal}} from a random float.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] genged opened a new pull request #13154: [FLINK-18704][table] Add DECIMAL type in datagen
genged opened a new pull request #13154: URL: https://github.com/apache/flink/pull/13154 ## What is the purpose of the change This pull request adds DECIMAL type (using BigDecimal) to the datagen Table source connector. ## Brief change log - Added decimalGenerator using BigDecimal - Added DECIMAL type in DataGenTableSourceFactory - Modified unit test to use DECIMAL ## Verifying this change This change added tests and can be verified as follows: - Modified tests to use DECIMAL ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: don't know - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13151: [FLINK-18943][python] Support CoMapFunction for Python DataStream API
flinkbot edited a comment on pull request #13151: URL: https://github.com/apache/flink/pull/13151#issuecomment-674350152 ## CI report: * 85ded7435e39bff6e7a54aa59b6314df8787bc91 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5546) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18879) Support Row Serialization and Deserialization schemas for DataStream source/sink
[ https://issues.apache.org/jira/browse/FLINK-18879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hequn Cheng closed FLINK-18879. --- Resolution: Resolved > Support Row Serialization and Deserialization schemas for DataStream > source/sink > > > Key: FLINK-18879 > URL: https://issues.apache.org/jira/browse/FLINK-18879 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Shuiqiang Chen >Assignee: Shuiqiang Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > There are many built-in RowSerializationSchemas and RowDeserializationSchemas > for DataStream Source and Sink, like JsonRow(De)SerializationSchema, > AvroRow(De)SerializationSchema. In Python DataStream API, we will also > support these schemas. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18879) Support Row Serialization and Deserialization schemas for DataStream source/sink
[ https://issues.apache.org/jira/browse/FLINK-18879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17178190#comment-17178190 ] Hequn Cheng commented on FLINK-18879: - Resolve in 1.12.0 via 1fc400d1f94a10d2f5b227cc8eaa1ac9754ec237 > Support Row Serialization and Deserialization schemas for DataStream > source/sink > > > Key: FLINK-18879 > URL: https://issues.apache.org/jira/browse/FLINK-18879 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Shuiqiang Chen >Assignee: Shuiqiang Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > There are many built-in RowSerializationSchemas and RowDeserializationSchemas > for DataStream Source and Sink, like JsonRow(De)SerializationSchema, > AvroRow(De)SerializationSchema. In Python DataStream API, we will also > support these schemas. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] hequn8128 merged pull request #13150: [FLINK-18879][python] Support Row Serialization and Deserialization s…
hequn8128 merged pull request #13150: URL: https://github.com/apache/flink/pull/13150 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13153: [FLINK-18966][python] Support key_by() on ConnectedStreams for Python DataStream API
flinkbot edited a comment on pull request #13153: URL: https://github.com/apache/flink/pull/13153#issuecomment-674367157 ## CI report: * f59963fa5ee56152ca93de32ed73e7fc67eb4250 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5549) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13152: [FLINK-18945][python] Support CoFlatMap for Python DataStream API
flinkbot edited a comment on pull request #13152: URL: https://github.com/apache/flink/pull/13152#issuecomment-674356260 ## CI report: * 1cc4bd5d29d7933c190df7d7476ff318c0dab7ee Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5547) * fb27e2e5684c59966cc505fbdc7969017ee067c9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5548) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13153: [FLINK-18966][python] Support key_by() on ConnectedStreams for Python DataStream API
flinkbot commented on pull request #13153: URL: https://github.com/apache/flink/pull/13153#issuecomment-674367157 ## CI report: * f59963fa5ee56152ca93de32ed73e7fc67eb4250 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13152: [FLINK-18945][python] Support CoFlatMap for Python DataStream API
flinkbot edited a comment on pull request #13152: URL: https://github.com/apache/flink/pull/13152#issuecomment-674356260 ## CI report: * 1cc4bd5d29d7933c190df7d7476ff318c0dab7ee Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5547) * fb27e2e5684c59966cc505fbdc7969017ee067c9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13153: [FLINK-18966][python] Support key_by() on ConnectedStreams for Python DataStream API
flinkbot commented on pull request #13153: URL: https://github.com/apache/flink/pull/13153#issuecomment-674365357 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit d0b550ef16bbd451819cf27eadc256348871325f (Sat Aug 15 07:59:29 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] hequn8128 commented on pull request #13153: [FLINK-18966][python] Support key_by() on ConnectedStreams for Python DataStream API
hequn8128 commented on pull request #13153: URL: https://github.com/apache/flink/pull/13153#issuecomment-674364816 CC @shuiqiangchen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18966) Support key_by() on ConnectedStreams for Python DataStream API
[ https://issues.apache.org/jira/browse/FLINK-18966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18966: --- Labels: pull-request-available (was: ) > Support key_by() on ConnectedStreams for Python DataStream API > -- > > Key: FLINK-18966 > URL: https://issues.apache.org/jira/browse/FLINK-18966 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Hequn Cheng >Assignee: Hequn Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] hequn8128 opened a new pull request #13153: [FLINK-18966][python] Support key_by() on ConnectedStreams for Python DataStream API
hequn8128 opened a new pull request #13153: URL: https://github.com/apache/flink/pull/13153 ## What is the purpose of the change This pull request supports key_by on Python `ConnectedStreams`. ## Brief change log - Add key_by on Python `ConnectedStreams`. - Refactor `DataStreamTwoInputPythonStatelessFunctionOperator` to support keyed `ConnectedStreams`. ## Verifying this change This change added tests and can be verified as follows: - Added `test_key_by_on_connect_stream` integration tests in test_data_stream ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (PythonDocs) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13152: [FLINK-18945][python] Support CoFlatMap for Python DataStream API
flinkbot edited a comment on pull request #13152: URL: https://github.com/apache/flink/pull/13152#issuecomment-674356260 ## CI report: * 1cc4bd5d29d7933c190df7d7476ff318c0dab7ee Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5547) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13152: [FLINK-18945][python] Support CoFlatMap for Python DataStream API
flinkbot edited a comment on pull request #13152: URL: https://github.com/apache/flink/pull/13152#issuecomment-674356260 ## CI report: * 1cc4bd5d29d7933c190df7d7476ff318c0dab7ee Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5547) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13152: [FLINK-18945][python] Support CoFlatMap for Python DataStream API
flinkbot commented on pull request #13152: URL: https://github.com/apache/flink/pull/13152#issuecomment-674356260 ## CI report: * 1cc4bd5d29d7933c190df7d7476ff318c0dab7ee UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13152: [FLINK-18945][python] Support CoFlatMap for Python DataStream API
flinkbot commented on pull request #13152: URL: https://github.com/apache/flink/pull/13152#issuecomment-674355686 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1cc4bd5d29d7933c190df7d7476ff318c0dab7ee (Sat Aug 15 06:12:24 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18945) Support CoFlatMap for Python DataStream API
[ https://issues.apache.org/jira/browse/FLINK-18945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18945: --- Labels: pull-request-available (was: ) > Support CoFlatMap for Python DataStream API > --- > > Key: FLINK-18945 > URL: https://issues.apache.org/jira/browse/FLINK-18945 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Hequn Cheng >Assignee: Hequn Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] hequn8128 opened a new pull request #13152: [FLINK-18945][python] Support CoFlatMap for Python DataStream API
hequn8128 opened a new pull request #13152: URL: https://github.com/apache/flink/pull/13152 ## What is the purpose of the change This pull request adds CoFlatMapFunction for Python DataStream API. ## Brief change log - Add CoFlatMapFunction on DataStream. ## Verifying this change This change added tests and can be verified as follows: - Added `test_co_flat_map_function_without_data_types` and `test_co_flat_map_function_with_data_types` integration tests in test_data_stream ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (PythonDocs) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
