[jira] [Commented] (FLINK-7) [GitHub] Enable Range Partitioner
[ https://issues.apache.org/jira/browse/FLINK-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050323#comment-15050323 ] ASF GitHub Bot commented on FLINK-7: Github user ChengXiangLi commented on the pull request: https://github.com/apache/flink/pull/1255#issuecomment-163533029 Thanks, @fhueske . I've updated the PR to use `PARTITION_CUSTOM`. The fail test should be unrelated. > [GitHub] Enable Range Partitioner > - > > Key: FLINK-7 > URL: https://issues.apache.org/jira/browse/FLINK-7 > Project: Flink > Issue Type: Sub-task > Components: Distributed Runtime >Reporter: GitHub Import >Assignee: Chengxiang Li > Fix For: pre-apache > > > The range partitioner is currently disabled. We need to implement the > following aspects: > 1) Distribution information, if available, must be propagated back together > with the ordering property. > 2) A generic bucket lookup structure (currently specific to PactRecord). > Tests to re-enable after fixing this issue: > - TeraSortITCase > - GlobalSortingITCase > - GlobalSortingMixedOrderITCase > Imported from GitHub > Url: https://github.com/stratosphere/stratosphere/issues/7 > Created by: [StephanEwen|https://github.com/StephanEwen] > Labels: core, enhancement, optimizer, > Milestone: Release 0.4 > Assignee: [fhueske|https://github.com/fhueske] > Created at: Fri Apr 26 13:48:24 CEST 2013 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-7] [Runtime] Enable Range Partitioner.
Github user ChengXiangLi commented on the pull request: https://github.com/apache/flink/pull/1255#issuecomment-163533029 Thanks, @fhueske . I've updated the PR to use `PARTITION_CUSTOM`. The fail test should be unrelated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3121) Watermark forwarding does not work for sources not producing any data
[ https://issues.apache.org/jira/browse/FLINK-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050459#comment-15050459 ] ASF GitHub Bot commented on FLINK-3121: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1447#issuecomment-16382 +1 for the Kafka part. > Watermark forwarding does not work for sources not producing any data > - > > Key: FLINK-3121 > URL: https://issues.apache.org/jira/browse/FLINK-3121 > Project: Flink > Issue Type: Bug >Reporter: Robert Metzger >Assignee: Aljoscha Krettek >Priority: Blocker > > This mailing list discussion explains the issue in detail: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-TimestampExtractor-and-FlinkKafkaConsumer082-td3488.html > As a workaround for now, the Kafka source can emit a final watermark for > sources not producing any data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3074) Make ApplicationMaster/JobManager akka port configurable
[ https://issues.apache.org/jira/browse/FLINK-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050456#comment-15050456 ] ASF GitHub Bot commented on FLINK-3074: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1416#issuecomment-163554565 Thank you for the review! I've merged it. > Make ApplicationMaster/JobManager akka port configurable > > > Key: FLINK-3074 > URL: https://issues.apache.org/jira/browse/FLINK-3074 > Project: Flink > Issue Type: Improvement > Components: YARN Client >Reporter: Robert Metzger >Assignee: Robert Metzger > Fix For: 1.0.0 > > > Similar to the BlobServer, the YARN ApplicationMaster should allow starting > it on a specified list or range of ports. > In cases where only certain ports are allowed by a firewall, users can > specify a range of ports where they want the AM to allocate its RPC port -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3074] Add config option to start YARN A...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1416#issuecomment-163554565 Thank you for the review! I've merged it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-3158) Shading does not remove google guava from flink-dist fat jar
Robert Metzger created FLINK-3158: - Summary: Shading does not remove google guava from flink-dist fat jar Key: FLINK-3158 URL: https://issues.apache.org/jira/browse/FLINK-3158 Project: Flink Issue Type: Bug Affects Versions: 0.10.1, 1.0.0 Reporter: Robert Metzger Assignee: Robert Metzger Priority: Blocker It seems that guava somehow slipped our checks and made it into the flink-dist fat jar again. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2769) Web dashboard port not configurable on client side
[
https://issues.apache.org/jira/browse/FLINK-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051605#comment-15051605
]
ASF GitHub Bot commented on FLINK-2769:
---
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1449#discussion_r47281222
--- Diff:
flink-runtime-web/web-dashboard/app/scripts/modules/overview/overview.svc.coffee
---
@@ -24,7 +24,7 @@ angular.module('flinkApp')
@loadOverview = ->
deferred = $q.defer()
-$http.get("overview")
+$http.get(flinkConfig.jobServer + "/overview")
--- End diff --
absolute url's (starting with '/') will break the YARN proxy stuff.
> Web dashboard port not configurable on client side
> --
>
> Key: FLINK-2769
> URL: https://issues.apache.org/jira/browse/FLINK-2769
> Project: Flink
> Issue Type: Bug
> Components: Webfrontend
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>Priority: Critical
> Fix For: 0.10.0
>
>
> The new web dashboard port configuration only affects the server side, but
> not the client side.
> To reproduce set in {{flink-conf.yaml}}:
> {code}
> jobmanager.web.port: 9091
> jobmanager.new-web-frontend: true
> {code}
> Run
> {code}
> $ bin/start-cluster.sh
> {code}
> You can browse to {{http://localhost:9091}}, but you won't see any data from
> the job manager. Requests to http://localhost:9091/overview etc. work as
> expected, but the client side JavaScript is running requests against
> {{http://localhost:8081}}.
> The problem is that
> {{flink-runtime-web/web-dashboard/app/scripts/index.coffee}} hard codes:
> {code}
> .value 'flinkConfig', {
> jobServer: 'http://localhost:8081'
> }
> {code} which is picked up by all generated scripts.
> This needs to be configurable for both standalone mode but especially for
> recovery mode. This was a major pain point for me today, because I was
> working on the dashboard and thought that my changes were wrong.
> In general, we need to add more tests to the new dashboard, which should soon
> replace the old one imo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2769] [runtime-web] Add configurable jo...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1449#discussion_r47281222
--- Diff:
flink-runtime-web/web-dashboard/app/scripts/modules/overview/overview.svc.coffee
---
@@ -24,7 +24,7 @@ angular.module('flinkApp')
@loadOverview = ->
deferred = $q.defer()
-$http.get("overview")
+$http.get(flinkConfig.jobServer + "/overview")
--- End diff --
absolute url's (starting with '/') will break the YARN proxy stuff.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2769] [runtime-web] Add configurable jo...
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/1449#discussion_r47291227
--- Diff:
flink-runtime-web/web-dashboard/app/scripts/modules/overview/overview.svc.coffee
---
@@ -24,7 +24,7 @@ angular.module('flinkApp')
@loadOverview = ->
deferred = $q.defer()
-$http.get("overview")
+$http.get(flinkConfig.jobServer + "/overview")
--- End diff --
Good catch. Was not on purpose
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-2769) Web dashboard port not configurable on client side
[
https://issues.apache.org/jira/browse/FLINK-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051707#comment-15051707
]
ASF GitHub Bot commented on FLINK-2769:
---
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/1449#discussion_r47291227
--- Diff:
flink-runtime-web/web-dashboard/app/scripts/modules/overview/overview.svc.coffee
---
@@ -24,7 +24,7 @@ angular.module('flinkApp')
@loadOverview = ->
deferred = $q.defer()
-$http.get("overview")
+$http.get(flinkConfig.jobServer + "/overview")
--- End diff --
Good catch. Was not on purpose
> Web dashboard port not configurable on client side
> --
>
> Key: FLINK-2769
> URL: https://issues.apache.org/jira/browse/FLINK-2769
> Project: Flink
> Issue Type: Bug
> Components: Webfrontend
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>Priority: Critical
> Fix For: 0.10.0
>
>
> The new web dashboard port configuration only affects the server side, but
> not the client side.
> To reproduce set in {{flink-conf.yaml}}:
> {code}
> jobmanager.web.port: 9091
> jobmanager.new-web-frontend: true
> {code}
> Run
> {code}
> $ bin/start-cluster.sh
> {code}
> You can browse to {{http://localhost:9091}}, but you won't see any data from
> the job manager. Requests to http://localhost:9091/overview etc. work as
> expected, but the client side JavaScript is running requests against
> {{http://localhost:8081}}.
> The problem is that
> {{flink-runtime-web/web-dashboard/app/scripts/index.coffee}} hard codes:
> {code}
> .value 'flinkConfig', {
> jobServer: 'http://localhost:8081'
> }
> {code} which is picked up by all generated scripts.
> This needs to be configurable for both standalone mode but especially for
> recovery mode. This was a major pain point for me today, because I was
> working on the dashboard and thought that my changes were wrong.
> In general, we need to add more tests to the new dashboard, which should soon
> replace the old one imo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2769) Web dashboard port not configurable on client side
[
https://issues.apache.org/jira/browse/FLINK-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050897#comment-15050897
]
ASF GitHub Bot commented on FLINK-2769:
---
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/1449#issuecomment-163594437
Thanks! It was 2769 and not 2796. Updated the PR.
> Web dashboard port not configurable on client side
> --
>
> Key: FLINK-2769
> URL: https://issues.apache.org/jira/browse/FLINK-2769
> Project: Flink
> Issue Type: Bug
> Components: Webfrontend
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>Priority: Critical
> Fix For: 0.10.0
>
>
> The new web dashboard port configuration only affects the server side, but
> not the client side.
> To reproduce set in {{flink-conf.yaml}}:
> {code}
> jobmanager.web.port: 9091
> jobmanager.new-web-frontend: true
> {code}
> Run
> {code}
> $ bin/start-cluster.sh
> {code}
> You can browse to {{http://localhost:9091}}, but you won't see any data from
> the job manager. Requests to http://localhost:9091/overview etc. work as
> expected, but the client side JavaScript is running requests against
> {{http://localhost:8081}}.
> The problem is that
> {{flink-runtime-web/web-dashboard/app/scripts/index.coffee}} hard codes:
> {code}
> .value 'flinkConfig', {
> jobServer: 'http://localhost:8081'
> }
> {code} which is picked up by all generated scripts.
> This needs to be configurable for both standalone mode but especially for
> recovery mode. This was a major pain point for me today, because I was
> working on the dashboard and thought that my changes were wrong.
> In general, we need to add more tests to the new dashboard, which should soon
> replace the old one imo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3158) Shading does not remove google guava from flink-dist fat jar
[
https://issues.apache.org/jira/browse/FLINK-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050615#comment-15050615
]
Robert Metzger commented on FLINK-3158:
---
There is a check in our travis build to ensure guava is not part of the build.
However, the check didn't work.
I think this is a change in the behavior of maven between Maven 3.2 and 3.3.
I've identified the following behavior:
{code}
mvn 3.2.5 (travis)
1. build ---> correct
2. build (flink-dist only) --> correct
mvn 3.3.9 (local, the machine used to create releases)
1. build --> incorrect
2. build (flink-dist only) --> correct
{code}
I think I have to contact the maven mailing list to understand whats going on.
Guava is not in the dependency tree of "flink-dist".
> Shading does not remove google guava from flink-dist fat jar
>
>
> Key: FLINK-3158
> URL: https://issues.apache.org/jira/browse/FLINK-3158
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.0.0, 0.10.1
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Blocker
>
> It seems that guava somehow slipped our checks and made it into the
> flink-dist fat jar again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2845) TimestampITCase.testWatermarkPropagation
[
https://issues.apache.org/jira/browse/FLINK-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050893#comment-15050893
]
Ufuk Celebi commented on FLINK-2845:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/95931340/log.txt
> TimestampITCase.testWatermarkPropagation
>
>
> Key: FLINK-2845
> URL: https://issues.apache.org/jira/browse/FLINK-2845
> Project: Flink
> Issue Type: Bug
> Components: Streaming
>Reporter: Matthias J. Sax
>Assignee: Aljoscha Krettek
>Priority: Critical
> Labels: test-stability
>
> https://travis-ci.org/apache/flink/jobs/84479912
> {noformat}
> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 16.592 sec
> <<< FAILURE! - in org.apache.flink.streaming.timestamp.TimestampITCase
> testWatermarkPropagation(org.apache.flink.streaming.timestamp.TimestampITCase)
> Time elapsed: 3.747 sec <<< FAILURE!
> java.lang.AssertionError: Wrong watermark.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.streaming.timestamp.TimestampITCase.testWatermarkPropagation(TimestampITCase.java:163)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2796] [runtime-web] Add configurable jo...
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1449#issuecomment-163570155 Wrong JIRA issue? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2796) CLI -q flag to supress the output does not work
[
https://issues.apache.org/jira/browse/FLINK-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050546#comment-15050546
]
ASF GitHub Bot commented on FLINK-2796:
---
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/1449#issuecomment-163570155
Wrong JIRA issue?
> CLI -q flag to supress the output does not work
> ---
>
> Key: FLINK-2796
> URL: https://issues.apache.org/jira/browse/FLINK-2796
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.10.0
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Labels: starter
> Fix For: 0.10.0
>
>
> The log output is shown regardless of whether -q is specified:
> {noformat}
> /bin/flink run -q examples/WordCount.jar
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-3156) FlinkKafkaConsumer fails with NPE on notifyCheckpointComplete
[
https://issues.apache.org/jira/browse/FLINK-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger resolved FLINK-3156.
---
Resolution: Fixed
Fix Version/s: 1.0.0
Fixed for 1.0 in http://git-wip-us.apache.org/repos/asf/flink/commit/c4a2d60c
> FlinkKafkaConsumer fails with NPE on notifyCheckpointComplete
> -
>
> Key: FLINK-3156
> URL: https://issues.apache.org/jira/browse/FLINK-3156
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Robert Metzger
> Fix For: 1.0.0
>
>
> The {{FlinkKafkaConsumer}} fails with a {{NullPointerException}} when a
> checkpoint was completed. The stack trace is
> {code}
> java.lang.RuntimeException: Error while confirming checkpoint
> at org.apache.flink.runtime.taskmanager.Task$2.run(Task.java:908)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.commitOffsets(FlinkKafkaConsumer.java:664)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.notifyCheckpointComplete(FlinkKafkaConsumer.java:578)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.notifyOfCompletedCheckpoint(AbstractUdfStreamOperator.java:176)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:546)
> at org.apache.flink.runtime.taskmanager.Task$2.run(Task.java:904)
> ... 5 more
> {code}
> Apparently, one of the {{KafkaTopicPartitionLeaders}} from the
> {{subscribedPartitions}} was {{null}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2769] [runtime-web] Add configurable jo...
Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1449#issuecomment-163866658 Tested it out locally. Works well with `gulp watch` and custom web frontend port. A few things I noticed: 1. Job Manager's log and stdout are not accessible with CORS, since they're served via `StaticFileServerHandler` which doesn't set CORS headers. This however shouldn't be an issue. We need to have a better logs service anyway, for both Job manager and task managers IMO. 2. At some point, we will remove the `GET /jobs//yarn-cancel` method, which is only there due to a limitation of yarn. To this end, there needs to be an `OPTIONS` handlers for the `DELETE` request to send CORS response headers in the preflight phase. https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS#Preflighted_requests. Not for the foreseeable future though. :-') --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2769) Web dashboard port not configurable on client side
[
https://issues.apache.org/jira/browse/FLINK-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15052359#comment-15052359
]
ASF GitHub Bot commented on FLINK-2769:
---
Github user sachingoel0101 commented on the pull request:
https://github.com/apache/flink/pull/1449#issuecomment-163866658
Tested it out locally. Works well with `gulp watch` and custom web frontend
port.
A few things I noticed:
1. Job Manager's log and stdout are not accessible with CORS, since they're
served via `StaticFileServerHandler` which doesn't set CORS headers. This
however shouldn't be an issue. We need to have a better logs service anyway,
for both Job manager and task managers IMO.
2. At some point, we will remove the `GET /jobs//yarn-cancel`
method, which is only there due to a limitation of yarn. To this end, there
needs to be an `OPTIONS` handlers for the `DELETE` request to send CORS
response headers in the preflight phase.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS#Preflighted_requests.
Not for the foreseeable future though. :-')
> Web dashboard port not configurable on client side
> --
>
> Key: FLINK-2769
> URL: https://issues.apache.org/jira/browse/FLINK-2769
> Project: Flink
> Issue Type: Bug
> Components: Webfrontend
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>Priority: Critical
> Fix For: 0.10.0
>
>
> The new web dashboard port configuration only affects the server side, but
> not the client side.
> To reproduce set in {{flink-conf.yaml}}:
> {code}
> jobmanager.web.port: 9091
> jobmanager.new-web-frontend: true
> {code}
> Run
> {code}
> $ bin/start-cluster.sh
> {code}
> You can browse to {{http://localhost:9091}}, but you won't see any data from
> the job manager. Requests to http://localhost:9091/overview etc. work as
> expected, but the client side JavaScript is running requests against
> {{http://localhost:8081}}.
> The problem is that
> {{flink-runtime-web/web-dashboard/app/scripts/index.coffee}} hard codes:
> {code}
> .value 'flinkConfig', {
> jobServer: 'http://localhost:8081'
> }
> {code} which is picked up by all generated scripts.
> This needs to be configurable for both standalone mode but especially for
> recovery mode. This was a major pain point for me today, because I was
> working on the dashboard and thought that my changes were wrong.
> In general, we need to add more tests to the new dashboard, which should soon
> replace the old one imo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3159) YARN AM should shutdown itself after a the job for which the AM was started has finished
Robert Metzger created FLINK-3159: - Summary: YARN AM should shutdown itself after a the job for which the AM was started has finished Key: FLINK-3159 URL: https://issues.apache.org/jira/browse/FLINK-3159 Project: Flink Issue Type: Improvement Affects Versions: 0.10.1 Reporter: Robert Metzger Assignee: Robert Metzger A user reported issues with Flink on YARN not shutting down after a job has finished on a single-job YARN cluster. We have already the APIs in place to stop the AM after a job has finished for detached per job yarn clusters. As an additional safety measure, I would like to register the shutdown also for attached per job yarn clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3097) Add support for custom functions in Table API
[
https://issues.apache.org/jira/browse/FLINK-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051201#comment-15051201
]
ASF GitHub Bot commented on FLINK-3097:
---
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1429#issuecomment-163680178
This looks good. :+1: Can you also integrate the support for Scala, i.e.
extract it from #1237
My only concern is that it introduces yet another instance where
TypeInformation is shipped to the cluster. I think we should tackle the problem
at some point. Maybe generate the code on the client and ship the code as a
string. Then use Janino to just compile it to a final class.
> Add support for custom functions in Table API
> -
>
> Key: FLINK-3097
> URL: https://issues.apache.org/jira/browse/FLINK-3097
> Project: Flink
> Issue Type: New Feature
> Components: Table API
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Currently, the Table API has a very limited set of built-in functions.
> Support for custom functions can solve this problem. Adding of a custom row
> function could look like:
> {code}
> TableEnvironment tableEnv = new TableEnvironment();
> RowFunction rf = new RowFunction() {
> @Override
> public String call(Object[] args) {
> return ((String) args[0]).trim();
> }
> };
> tableEnv.getConfig().registerRowFunction("TRIM", rf,
> BasicTypeInfo.STRING_TYPE_INFO);
> DataSource input = env.fromElements(
> new Tuple1<>(" 1 "));
> Table table = tableEnv.fromDataSet(input);
> Table result = table.select("TRIM(f0)");
> {code}
> This feature is also necessary as part of FLINK-2099.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3097] [table] Add support for custom fu...
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1429#issuecomment-163680178 This looks good. :+1: Can you also integrate the support for Scala, i.e. extract it from #1237 My only concern is that it introduces yet another instance where TypeInformation is shipped to the cluster. I think we should tackle the problem at some point. Maybe generate the code on the client and ship the code as a string. Then use Janino to just compile it to a final class. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
