[jira] [Work logged] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[
https://issues.apache.org/jira/browse/BEAM-9093?focusedWorklogId=377109&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377109
]
ASF GitHub Bot logged work on BEAM-9093:
Author: ASF GitHub Bot
Created on: 25/Jan/20 00:25
Start Date: 25/Jan/20 00:25
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10613: [BEAM-9093]
Log invalid overwrites in pipeline options
URL: https://github.com/apache/beam/pull/10613
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 377109)
Time Spent: 1.5h (was: 1h 20m)
> Pipeline options which with different underlying store variable does not get
> over written
> -
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[
https://issues.apache.org/jira/browse/BEAM-9093?focusedWorklogId=376586&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376586
]
ASF GitHub Bot logged work on BEAM-9093:
Author: ASF GitHub Bot
Created on: 23/Jan/20 21:28
Start Date: 23/Jan/20 21:28
Worklog Time Spent: 10m
Work Description: angoenka commented on issue #10613: [BEAM-9093] Log
invalid overwrites in pipeline options
URL: https://github.com/apache/beam/pull/10613#issuecomment-577885505
Run Python2_PVR_Flink PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 376586)
Time Spent: 1h 20m (was: 1h 10m)
> Pipeline options which with different underlying store variable does not get
> over written
> -
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[
https://issues.apache.org/jira/browse/BEAM-9093?focusedWorklogId=376466&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376466
]
ASF GitHub Bot logged work on BEAM-9093:
Author: ASF GitHub Bot
Created on: 23/Jan/20 19:34
Start Date: 23/Jan/20 19:34
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10613: [BEAM-9093]
Log invalid overwrites in pipeline options
URL: https://github.com/apache/beam/pull/10613#discussion_r370313458
##
File path: sdks/python/apache_beam/options/pipeline_options_test.py
##
@@ -264,6 +264,18 @@ def test_override_options(self):
self.assertEqual(options.get_all_options()['num_workers'], 5)
self.assertTrue(options.get_all_options()['mock_flag'])
+ def test_override_init_options(self):
+base_flags = ['--num_workers', '5']
+options = PipelineOptions(base_flags, mock_flag=True)
+self.assertEqual(options.get_all_options()['num_workers'], 5)
+self.assertEqual(options.get_all_options()['mock_flag'], True)
+
+ def test_invalid_override_init_options(self):
+base_flags = ['--num_workers', '5']
+options = PipelineOptions(base_flags, mock_invalid_flag=True)
+self.assertEqual(options.get_all_options()['num_workers'], 5)
+self.assertEqual(options.get_all_options()['mock_flag'], False)
Review comment:
Checking log message for test correctness will not be idle i suppose.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 376466)
Time Spent: 1h 10m (was: 1h)
> Pipeline options which with different underlying store variable does not get
> over written
> -
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[
https://issues.apache.org/jira/browse/BEAM-9093?focusedWorklogId=376458&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376458
]
ASF GitHub Bot logged work on BEAM-9093:
Author: ASF GitHub Bot
Created on: 23/Jan/20 19:33
Start Date: 23/Jan/20 19:33
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10613: [BEAM-9093]
Log invalid overwrites in pipeline options
URL: https://github.com/apache/beam/pull/10613#discussion_r370313458
##
File path: sdks/python/apache_beam/options/pipeline_options_test.py
##
@@ -264,6 +264,18 @@ def test_override_options(self):
self.assertEqual(options.get_all_options()['num_workers'], 5)
self.assertTrue(options.get_all_options()['mock_flag'])
+ def test_override_init_options(self):
+base_flags = ['--num_workers', '5']
+options = PipelineOptions(base_flags, mock_flag=True)
+self.assertEqual(options.get_all_options()['num_workers'], 5)
+self.assertEqual(options.get_all_options()['mock_flag'], True)
+
+ def test_invalid_override_init_options(self):
+base_flags = ['--num_workers', '5']
+options = PipelineOptions(base_flags, mock_invalid_flag=True)
+self.assertEqual(options.get_all_options()['num_workers'], 5)
+self.assertEqual(options.get_all_options()['mock_flag'], False)
Review comment:
Checking logging message for test correctness will not be idle i suppose.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 376458)
Time Spent: 50m (was: 40m)
> Pipeline options which with different underlying store variable does not get
> over written
> -
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[
https://issues.apache.org/jira/browse/BEAM-9093?focusedWorklogId=376459&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376459
]
ASF GitHub Bot logged work on BEAM-9093:
Author: ASF GitHub Bot
Created on: 23/Jan/20 19:33
Start Date: 23/Jan/20 19:33
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10613: [BEAM-9093]
Log invalid overwrites in pipeline options
URL: https://github.com/apache/beam/pull/10613#discussion_r370313565
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -288,15 +288,20 @@ def get_all_options(self,
_LOGGER.warning("Discarding unparseable args: %s", unknown_args)
result = vars(known_args)
+overrides = self._all_options.copy()
# Apply the overrides if any
for k in list(result):
+ overrides.pop(k, None)
if k in self._all_options:
result[k] = self._all_options[k]
if (drop_default and
parser.get_default(k) == result[k] and
not isinstance(parser.get_default(k), ValueProvider)):
del result[k]
+if overrides:
Review comment:
Good catch. I don't know why lint did not catch it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 376459)
Time Spent: 1h (was: 50m)
> Pipeline options which with different underlying store variable does not get
> over written
> -
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[
https://issues.apache.org/jira/browse/BEAM-9093?focusedWorklogId=375240&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-375240
]
ASF GitHub Bot logged work on BEAM-9093:
Author: ASF GitHub Bot
Created on: 21/Jan/20 21:59
Start Date: 21/Jan/20 21:59
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10613: [BEAM-9093] Log
invalid overwrites in pipeline options
URL: https://github.com/apache/beam/pull/10613#discussion_r369264919
##
File path: sdks/python/apache_beam/options/pipeline_options_test.py
##
@@ -264,6 +264,18 @@ def test_override_options(self):
self.assertEqual(options.get_all_options()['num_workers'], 5)
self.assertTrue(options.get_all_options()['mock_flag'])
+ def test_override_init_options(self):
+base_flags = ['--num_workers', '5']
+options = PipelineOptions(base_flags, mock_flag=True)
+self.assertEqual(options.get_all_options()['num_workers'], 5)
+self.assertEqual(options.get_all_options()['mock_flag'], True)
+
+ def test_invalid_override_init_options(self):
+base_flags = ['--num_workers', '5']
+options = PipelineOptions(base_flags, mock_invalid_flag=True)
+self.assertEqual(options.get_all_options()['num_workers'], 5)
+self.assertEqual(options.get_all_options()['mock_flag'], False)
Review comment:
It would be great if we could also capture the logging in addition to only
checking `mock_flag` is untouched.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 375240)
Time Spent: 40m (was: 0.5h)
> Pipeline options which with different underlying store variable does not get
> over written
> -
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[
https://issues.apache.org/jira/browse/BEAM-9093?focusedWorklogId=375241&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-375241
]
ASF GitHub Bot logged work on BEAM-9093:
Author: ASF GitHub Bot
Created on: 21/Jan/20 21:59
Start Date: 21/Jan/20 21:59
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10613: [BEAM-9093] Log
invalid overwrites in pipeline options
URL: https://github.com/apache/beam/pull/10613#discussion_r369260182
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -288,15 +288,20 @@ def get_all_options(self,
_LOGGER.warning("Discarding unparseable args: %s", unknown_args)
result = vars(known_args)
+overrides = self._all_options.copy()
# Apply the overrides if any
for k in list(result):
+ overrides.pop(k, None)
if k in self._all_options:
result[k] = self._all_options[k]
if (drop_default and
parser.get_default(k) == result[k] and
not isinstance(parser.get_default(k), ValueProvider)):
del result[k]
+if overrides:
Review comment:
Looks like there are two spaces.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 375241)
Time Spent: 40m (was: 0.5h)
> Pipeline options which with different underlying store variable does not get
> over written
> -
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[
https://issues.apache.org/jira/browse/BEAM-9093?focusedWorklogId=375177&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-375177
]
ASF GitHub Bot logged work on BEAM-9093:
Author: ASF GitHub Bot
Created on: 21/Jan/20 19:28
Start Date: 21/Jan/20 19:28
Worklog Time Spent: 10m
Work Description: angoenka commented on issue #10613: [BEAM-9093] Log
invalid overwrites in pipeline options
URL: https://github.com/apache/beam/pull/10613#issuecomment-576841292
Friendly ping!
@robertwb @ihji
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 375177)
Time Spent: 0.5h (was: 20m)
> Pipeline options which with different underlying store variable does not get
> over written
> -
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[
https://issues.apache.org/jira/browse/BEAM-9093?focusedWorklogId=373241&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373241
]
ASF GitHub Bot logged work on BEAM-9093:
Author: ASF GitHub Bot
Created on: 16/Jan/20 20:02
Start Date: 16/Jan/20 20:02
Worklog Time Spent: 10m
Work Description: angoenka commented on issue #10613: [BEAM-9093] Log
invalid overwrites in pipeline options
URL: https://github.com/apache/beam/pull/10613#issuecomment-575321578
R: @robertwb @ihji
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373241)
Time Spent: 20m (was: 10m)
> Pipeline options which with different underlying store variable does not get
> over written
> -
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[ https://issues.apache.org/jira/browse/BEAM-9093?focusedWorklogId=373240&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373240 ] ASF GitHub Bot logged work on BEAM-9093: Author: ASF GitHub Bot Created on: 16/Jan/20 20:02 Start Date: 16/Jan/20 20:02 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10613: [BEAM-9093] Log invalid overwrites in pipeline options URL: https://github.com/apache/beam/pull/10613 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[