[jira] [Commented] (FLINK-9151) standalone cluster scripts should pass FLINK_CONF_DIR to job manager and task managers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435145#comment-16435145
 ] 

ASF GitHub Bot commented on FLINK-9151:
---

Github user facboy commented on the issue:

https://github.com/apache/flink/pull/5831
  
Actually this can be handled by `env.ssh.opts` provided your target servers 
allow you to send the `FLINK_CONF_DIR` env var in an ssh session.


> standalone cluster scripts should pass FLINK_CONF_DIR to job manager and task 
> managers
> --
>
> Key: FLINK-9151
> URL: https://issues.apache.org/jira/browse/FLINK-9151
> Project: Flink
>  Issue Type: Improvement
>  Components: Startup Shell Scripts
>Affects Versions: 1.4.1
>Reporter: Christopher Ng
>Priority: Minor
>
> At the moment FLINK_CONF_DIR is not passed to the job manager and task 
> manager when they are started over SSH.  This means that if the user has a 
> locally set FLINK_CONF_DIR that is not configured by their login shell, it is 
> not used by the launched job manager and task manager which can result in 
> silently failing to launch if there are errors due to Flink not using the 
> correct config dir.
> One particular inconsistency is that a TaskManagers may be launched locally 
> (without ssh) on localhost, but JobManagers are always launched over ssh.  In 
> my particular case this meant that the TaskManager launched but the 
> JobManager silently failed to launch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9151) standalone cluster scripts should pass FLINK_CONF_DIR to job manager and task managers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435146#comment-16435146
 ] 

ASF GitHub Bot commented on FLINK-9151:
---

Github user facboy closed the pull request at:

https://github.com/apache/flink/pull/5831


> standalone cluster scripts should pass FLINK_CONF_DIR to job manager and task 
> managers
> --
>
> Key: FLINK-9151
> URL: https://issues.apache.org/jira/browse/FLINK-9151
> Project: Flink
>  Issue Type: Improvement
>  Components: Startup Shell Scripts
>Affects Versions: 1.4.1
>Reporter: Christopher Ng
>Priority: Minor
>
> At the moment FLINK_CONF_DIR is not passed to the job manager and task 
> manager when they are started over SSH.  This means that if the user has a 
> locally set FLINK_CONF_DIR that is not configured by their login shell, it is 
> not used by the launched job manager and task manager which can result in 
> silently failing to launch if there are errors due to Flink not using the 
> correct config dir.
> One particular inconsistency is that a TaskManagers may be launched locally 
> (without ssh) on localhost, but JobManagers are always launched over ssh.  In 
> my particular case this meant that the TaskManager launched but the 
> JobManager silently failed to launch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9151) standalone cluster scripts should pass FLINK_CONF_DIR to job manager and task managers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435142#comment-16435142
 ] 

ASF GitHub Bot commented on FLINK-9151:
---

Github user facboy commented on the issue:

https://github.com/apache/flink/pull/5832
  
Actually this can be handled by `env.ssh.opts` provided your target servers 
allow you to send the `FLINK_CONF_DIR` env var in an ssh session.


> standalone cluster scripts should pass FLINK_CONF_DIR to job manager and task 
> managers
> --
>
> Key: FLINK-9151
> URL: https://issues.apache.org/jira/browse/FLINK-9151
> Project: Flink
>  Issue Type: Improvement
>  Components: Startup Shell Scripts
>Affects Versions: 1.4.1
>Reporter: Christopher Ng
>Priority: Minor
>
> At the moment FLINK_CONF_DIR is not passed to the job manager and task 
> manager when they are started over SSH.  This means that if the user has a 
> locally set FLINK_CONF_DIR that is not configured by their login shell, it is 
> not used by the launched job manager and task manager which can result in 
> silently failing to launch if there are errors due to Flink not using the 
> correct config dir.
> One particular inconsistency is that a TaskManagers may be launched locally 
> (without ssh) on localhost, but JobManagers are always launched over ssh.  In 
> my particular case this meant that the TaskManager launched but the 
> JobManager silently failed to launch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9151) standalone cluster scripts should pass FLINK_CONF_DIR to job manager and task managers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435144#comment-16435144
 ] 

ASF GitHub Bot commented on FLINK-9151:
---

Github user facboy closed the pull request at:

https://github.com/apache/flink/pull/5832


> standalone cluster scripts should pass FLINK_CONF_DIR to job manager and task 
> managers
> --
>
> Key: FLINK-9151
> URL: https://issues.apache.org/jira/browse/FLINK-9151
> Project: Flink
>  Issue Type: Improvement
>  Components: Startup Shell Scripts
>Affects Versions: 1.4.1
>Reporter: Christopher Ng
>Priority: Minor
>
> At the moment FLINK_CONF_DIR is not passed to the job manager and task 
> manager when they are started over SSH.  This means that if the user has a 
> locally set FLINK_CONF_DIR that is not configured by their login shell, it is 
> not used by the launched job manager and task manager which can result in 
> silently failing to launch if there are errors due to Flink not using the 
> correct config dir.
> One particular inconsistency is that a TaskManagers may be launched locally 
> (without ssh) on localhost, but JobManagers are always launched over ssh.  In 
> my particular case this meant that the TaskManager launched but the 
> JobManager silently failed to launch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9151) standalone cluster scripts should pass FLINK_CONF_DIR to job manager and task managers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431959#comment-16431959
 ] 

ASF GitHub Bot commented on FLINK-9151:
---

GitHub user facboy opened a pull request:

https://github.com/apache/flink/pull/5832

[FLINK-9151] [Startup Shell Scripts] Export FLINK_CONF_DIR to job manager 
and task managers in standalone cluster mode (master)

## What is the purpose of the change

This pull request makes the standalone cluster scripts pass FLINK_CONF_DIR 
to the launched job managers and task managers, rather than relying on the 
default config dir on the target host.

## Brief change log

- Added export FLINK_CONF_DIR to `config.sh` and `start_cluser.sh`

## Verifying this change

- I've only manually verified the change on 1.4.x.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): no
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
  - The serializers: no
  - The runtime per-record code paths (performance sensitive): no
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: yes
  - The S3 file system connector: no

## Documentation

  - Does this pull request introduce a new feature? no
  - If yes, how is the feature documented? not applicable


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/facboy/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5832.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5832


commit ec023753dadd1b4fda1b4ab23e9b0ce459f44667
Author: Christopher Ng 
Date:   2018-04-10T08:39:41Z

FLINK-9151 Export FLINK_CONF_DIR to job manager and task managers in 
standalone cluster mode.




> standalone cluster scripts should pass FLINK_CONF_DIR to job manager and task 
> managers
> --
>
> Key: FLINK-9151
> URL: https://issues.apache.org/jira/browse/FLINK-9151
> Project: Flink
>  Issue Type: Improvement
>  Components: Startup Shell Scripts
>Affects Versions: 1.4.1
>Reporter: Christopher Ng
>Priority: Minor
>
> At the moment FLINK_CONF_DIR is not passed to the job manager and task 
> manager when they are started over SSH.  This means that if the user has a 
> locally set FLINK_CONF_DIR that is not configured by their login shell, it is 
> not used by the launched job manager and task manager which can result in 
> silently failing to launch if there are errors due to Flink not using the 
> correct config dir.
> One particular inconsistency is that a TaskManagers may be launched locally 
> (without ssh) on localhost, but JobManagers are always launched over ssh.  In 
> my particular case this meant that the TaskManager launched but the 
> JobManager silently failed to launch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9151) standalone cluster scripts should pass FLINK_CONF_DIR to job manager and task managers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431948#comment-16431948
 ] 

ASF GitHub Bot commented on FLINK-9151:
---

GitHub user facboy opened a pull request:

https://github.com/apache/flink/pull/5831

[FLINK-9151] [Startup Shell Scripts] Export FLINK_CONF_DIR to job manager 
and task managers in standalone cluster mode

## What is the purpose of the change

This pull request makes the standalone cluster scripts pass FLINK_CONF_DIR 
to the launched job managers and task managers, rather than relying on the 
default config dir on the target host.

## Brief change log

- Added export FLINK_CONF_DIR to `config.sh` and `start_cluser.sh`

## Verifying this change

- Manually verified the change by running a standalone cluster with a local 
FLINK_CONF_DIR environment variable set.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): no
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
  - The serializers: no
  - The runtime per-record code paths (performance sensitive): no
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: yes
  - The S3 file system connector: no

## Documentation

  - Does this pull request introduce a new feature? no
  - If yes, how is the feature documented? not applicable


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/facboy/flink 1.4.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5831.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5831


commit 760d1a6bb75eb9519a4b93eb3cf34ad1605621da
Author: yew1eb 
Date:   2017-11-07T01:06:45Z

[hotfix][docs] Add type for numLateRecordsDropped metric in docs

commit 07830e7897a42b5d12f0b33c42933c6ca78e70d3
Author: zentol 
Date:   2017-11-07T11:16:04Z

[hotfix][rat] Add missing rat exclusions

Another set of RAT exclusions to prevent errors on Windows.

commit aab36f934548a5697c5c461b2a79c7cf3fd0d756
Author: kkloudas 
Date:   2017-11-06T11:43:18Z

[FLINK-7823][QS] Update Queryable State configuration parameters.

commit 819995454611be6a85e2933318d053b2c25a18f7
Author: kkloudas 
Date:   2017-11-06T16:21:45Z

[FLINK-7822][QS][doc] Update Queryable State docs.

commit 564c9934fd3aaba462a7415788b3d55486146f9b
Author: Aljoscha Krettek 
Date:   2017-11-07T17:27:16Z

[hotfix] Use correct commit id in GenericWriteAheadSink.notifyCheckpoint

commit 3cbf467ebdf639df4d7d4da78b7bc2929aa4b5d9
Author: Piotr Nowojski 
Date:   2017-11-06T13:03:16Z

[hotfix][kafka] Extract TransactionalIdsGenerator class from 
FlinkKafkaProducer011

This is pure refactor without any functional changes.

commit 460e27aeb5e246aff0f8137448441c315123608c
Author: Piotr Nowojski 
Date:   2017-11-06T13:14:01Z

[FLINK-7978][kafka] Ensure that transactional ids will never clash

Previously transactional ids to use and to abort could clash between
subtasks. This could lead to a race condition between initialization
and writting the data, where one subtask is still initializing/aborting
some transactional id while different subtask is already trying to write
the data using the same transactional id.

commit b677c8d69b81fb3594798ba2761fdb7e2edea5db
Author: Fabian Hueske 
Date:   2017-11-07T22:43:45Z

[hotfix] [docs] Improve Supported Types section of Table API & SQL docs.

commit dc1ca78a4e4cb339e9fbf0c90700f3204e091c53
Author: Fabian Hueske 
Date:   2017-11-07T23:12:49Z

[hotfix] [docs] Fix UDTF join description in SQL docs.

commit 5af710080eb72d23d8d2f6a77d1825f3d8a009ae
Author: zentol 
Date:   2017-11-07T10:40:15Z

[FLINK-8004][metrics][docs] Fix usage examples

commit 49dc380697627189f6ac2e8bf5a084ac85c21ed5
Author: zentol 
Date:   2017-11-07T14:36:49Z

[FLINK-8010][build] Bump remaining flink-shaded versions

commit 17aae5af4a7973348067d5786cd4f16fc9da2639
Author: Tzu-Li (Gordon) Tai 
Date:   2017-11-07T11:35:33Z

[FLINK-8001] [kafka] Prevent PeriodicWatermarkEmitter from violating IDLE 
status

Prior to this commit, a bug exists such that if a Kafka consumer subtask
initially marks itself as idle because it didn't have any partitions to
subscribe to, that idleness status will be violated when the
PeriodicWatermarkEmitter is fired.

The problem is that the