[jira] [Work logged] (BEAM-7547) StreamingDataflowWorker can observe inconsistent cache for stale work items
[ https://issues.apache.org/jira/browse/BEAM-7547?focusedWorklogId=269052=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-269052 ] ASF GitHub Bot logged work on BEAM-7547: Author: ASF GitHub Bot Created on: 28/Jun/19 05:16 Start Date: 28/Jun/19 05:16 Worklog Time Spent: 10m Work Description: scwhittle commented on issue #8842: [BEAM-7547] Avoid WindmillStateCache cache hits for stale work. URL: https://github.com/apache/beam/pull/8842#issuecomment-506603549 Run JavaPortabilityApi PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 269052) Time Spent: 2h 10m (was: 2h) > StreamingDataflowWorker can observe inconsistent cache for stale work items > --- > > Key: BEAM-7547 > URL: https://issues.apache.org/jira/browse/BEAM-7547 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Sam Whittle >Assignee: Sam Whittle >Priority: Minor > Time Spent: 2h 10m > Remaining Estimate: 0h > > 1. Dataflow backend generates a work item with a cache token C. > 2. StreamingDataflowWorker receives the work item and reads the state using > C, it either hits the cache or performs a read. > 3. Dataflow backend sends a retry of the work item (possibly because it > thinks original work item never reached the StreamingDataflowWorker). > 4. StreamingDataflowWorker commits the work item and gets ack from dataflow > backend. It caches the state for the key using C. > 5. StreamingDataflowWorker receives the retried work item with cache token C. > It uses the cached state and causes possible user consistency failures > because the cache view is of after the work item completed processing. > Note that this will not cause corrupted Dataflow persistent state because the > commit of the retried work item using the inconsistent cache will fail. > However it may cause failures in user logic for example if they keep the set > of all seen items in state and throw an exception on duplicates which should > have been removed by an upstream stage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268986=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268986 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 28/Jun/19 02:46 Start Date: 28/Jun/19 02:46 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506579569 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268986) Time Spent: 18h 10m (was: 18h) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 18h 10m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268985
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 28/Jun/19 02:46
Start Date: 28/Jun/19 02:46
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #8960: [BEAM-7548] Fix
flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8960#issuecomment-506579507
Run Portable_Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268985)
Time Spent: 5.5h (was: 5h 20m)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response =
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268980
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 28/Jun/19 02:11
Start Date: 28/Jun/19 02:11
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #8960: [BEAM-7548] Fix
flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8960#issuecomment-506573470
R: @chamikaramj
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268980)
Time Spent: 5h 20m (was: 5h 10m)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response =
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[ https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268979 ] ASF GitHub Bot logged work on BEAM-7548: Author: ASF GitHub Bot Created on: 28/Jun/19 02:07 Start Date: 28/Jun/19 02:07 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #8960: [BEAM-7548] Fix flaky tests for ApproximateUnique URL: https://github.com/apache/beam/pull/8960 Adding more retries and skip very flaky tests. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [![Build
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[
https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=268967=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268967
]
ASF GitHub Bot logged work on BEAM-7389:
Author: ASF GitHub Bot
Created on: 28/Jun/19 01:20
Start Date: 28/Jun/19 01:20
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8904: [BEAM-7389]
Add Python snippet for Partition transform
URL: https://github.com/apache/beam/pull/8904#discussion_r298423896
##
File path:
sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py
##
@@ -0,0 +1,87 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.transforms.element_wise.partition import *
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.element_wise.partition.print',
lambda elem: elem)
+# pylint: enable=line-too-long
+class PartitionTest(unittest.TestCase):
+ def __init__(self, methodName):
+super(PartitionTest, self).__init__(methodName)
+# [START partitions]
+annuals = [
+{'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+]
+biennials = [
+{'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+]
+perennials = [
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+]
+# [END partitions]
+
+def partitions_test(actual1, actual2, actual3):
+ assert_that(actual1, equal_to(annuals), label='assert annuals')
+ assert_that(actual2, equal_to(biennials), label='assert biennials')
+ assert_that(actual3, equal_to(perennials), label='assert perennials')
+self.partitions_test = partitions_test
+
+# [START train_test]
+train_dataset = [
Review comment:
1. I think we should we relax the requirement which elements fall into
`test_dataset` and which into `train_dataset` in this scenario. The transform's
semantic does not specify which elements of PCollection will be picked, since
PCollections don't have an order of elements, however the test assumes a
particular order, which may change when implementation of DirectRunner changes.
But we can still verify an expectation that 5 elements were split into 4 + 1
and altogether make up for the whole set of initial elements.
2. Even though we fix the seed for random, I think the generator
implementation may change in a future version of Python, and the sequence of
numbers with the same seed might be different on Python 3.7 and Python 3.9. How
about we stub out the values for `random()` via mock.patch?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268967)
Time Spent: 18h 20m (was: 18h 10m)
> Colab examples for element-wise transforms (Python)
> ---
>
> Key: BEAM-7389
> URL: https://issues.apache.org/jira/browse/BEAM-7389
> Project: Beam
> Issue Type: Improvement
> Components: website
>Reporter: Rose Nguyen
>Assignee: David Cavazos
>Priority: Minor
> Time Spent: 18h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[
https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=268968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268968
]
ASF GitHub Bot logged work on BEAM-7389:
Author: ASF GitHub Bot
Created on: 28/Jun/19 01:20
Start Date: 28/Jun/19 01:20
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8904: [BEAM-7389]
Add Python snippet for Partition transform
URL: https://github.com/apache/beam/pull/8904#discussion_r298341353
##
File path:
sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py
##
@@ -0,0 +1,130 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def partition_function(test=None):
+ # [START partition_function]
+ import apache_beam as beam
+
+ durations = ['annual', 'biennial', 'perennial']
+
+ def by_duration(plant, num_partitions):
+return durations.index(plant['duration'])
+
+ with beam.Pipeline() as pipeline:
+annuals, biennials, perennials = (
+pipeline
+| 'Gardening plants' >> beam.Create([
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+])
+| 'Partition' >> beam.Partition(by_duration, len(durations))
+)
+_ = (
+annuals
+| 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))
+)
+_ = (
+biennials
+| 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))
+)
+_ = (
+perennials
+| 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))
+)
+# [END partition_function]
+if test:
+ test(annuals, biennials, perennials)
+
+
+def partition_lambda(test=None):
+ # [START partition_lambda]
+ import apache_beam as beam
+
+ durations = ['annual', 'biennial', 'perennial']
+
+ with beam.Pipeline() as pipeline:
+annuals, biennials, perennials = (
+pipeline
+| 'Gardening plants' >> beam.Create([
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+])
+| 'Partition' >> beam.Partition(
+lambda plant, num_partitions: durations.index(plant['duration']),
+len(durations),
+)
+)
+_ = (
+annuals
+| 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))
+)
+_ = (
+biennials
+| 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))
+)
+_ = (
+perennials
+| 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))
+)
+# [END partition_lambda]
+if test:
+ test(annuals, biennials, perennials)
+
+
+def partition_multiple_arguments(test=None):
+ # [START partition_multiple_arguments]
+ import apache_beam as beam
+ from random import Random
+
+ random = Random(0)
+ with beam.Pipeline() as pipeline:
+train_dataset, test_dataset = (
+pipeline
+| 'Gardening plants' >> beam.Create([
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+])
+| 'Partition' >> beam.Partition(
+lambda plant, n, train_ratio: int(random.random() > train_ratio),
+2, train_ratio=0.8,
Review comment:
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[
https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=268966=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268966
]
ASF GitHub Bot logged work on BEAM-7389:
Author: ASF GitHub Bot
Created on: 28/Jun/19 01:20
Start Date: 28/Jun/19 01:20
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8904: [BEAM-7389]
Add Python snippet for Partition transform
URL: https://github.com/apache/beam/pull/8904#discussion_r298344639
##
File path:
sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py
##
@@ -0,0 +1,130 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def partition_function(test=None):
+ # [START partition_function]
+ import apache_beam as beam
+
+ durations = ['annual', 'biennial', 'perennial']
+
+ def by_duration(plant, num_partitions):
+return durations.index(plant['duration'])
+
+ with beam.Pipeline() as pipeline:
+annuals, biennials, perennials = (
+pipeline
+| 'Gardening plants' >> beam.Create([
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+])
+| 'Partition' >> beam.Partition(by_duration, len(durations))
+)
+_ = (
+annuals
+| 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))
+)
+_ = (
+biennials
+| 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))
+)
+_ = (
+perennials
+| 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))
+)
+# [END partition_function]
+if test:
+ test(annuals, biennials, perennials)
+
+
+def partition_lambda(test=None):
+ # [START partition_lambda]
+ import apache_beam as beam
+
+ durations = ['annual', 'biennial', 'perennial']
+
+ with beam.Pipeline() as pipeline:
+annuals, biennials, perennials = (
+pipeline
+| 'Gardening plants' >> beam.Create([
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+])
+| 'Partition' >> beam.Partition(
+lambda plant, num_partitions: durations.index(plant['duration']),
+len(durations),
+)
+)
+_ = (
+annuals
+| 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))
+)
+_ = (
+biennials
+| 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))
+)
+_ = (
+perennials
+| 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))
+)
+# [END partition_lambda]
+if test:
+ test(annuals, biennials, perennials)
+
+
+def partition_multiple_arguments(test=None):
+ # [START partition_multiple_arguments]
+ import apache_beam as beam
+ from random import Random
+
+ random = Random(0)
+ with beam.Pipeline() as pipeline:
+train_dataset, test_dataset = (
+pipeline
+| 'Gardening plants' >> beam.Create([
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+])
+| 'Partition' >> beam.Partition(
+lambda plant, n, train_ratio: int(random.random() > train_ratio),
Review comment:
This lambda feels a little hacky
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[
https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=268964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268964
]
ASF GitHub Bot logged work on BEAM-7389:
Author: ASF GitHub Bot
Created on: 28/Jun/19 01:20
Start Date: 28/Jun/19 01:20
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8904: [BEAM-7389]
Add Python snippet for Partition transform
URL: https://github.com/apache/beam/pull/8904#discussion_r298421268
##
File path:
sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py
##
@@ -0,0 +1,87 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.transforms.element_wise.partition import *
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.element_wise.partition.print',
lambda elem: elem)
+# pylint: enable=line-too-long
+class PartitionTest(unittest.TestCase):
+ def __init__(self, methodName):
+super(PartitionTest, self).__init__(methodName)
+# [START partitions]
+annuals = [
+{'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+]
+biennials = [
+{'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+]
+perennials = [
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+]
+# [END partitions]
+
+def partitions_test(actual1, actual2, actual3):
Review comment:
How about we move test helpers (`partitions_test` etc) and test lists
(`annuals` etc) outside of `PartititionTest`? This way don't need to
`self.partitions_test = partitions_test` and can just pass:
```
def test_partition_function(self):
partition_function(partitions_test)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268964)
Time Spent: 17h 50m (was: 17h 40m)
> Colab examples for element-wise transforms (Python)
> ---
>
> Key: BEAM-7389
> URL: https://issues.apache.org/jira/browse/BEAM-7389
> Project: Beam
> Issue Type: Improvement
> Components: website
>Reporter: Rose Nguyen
>Assignee: David Cavazos
>Priority: Minor
> Time Spent: 17h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[ https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=268965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268965 ] ASF GitHub Bot logged work on BEAM-7389: Author: ASF GitHub Bot Created on: 28/Jun/19 01:20 Start Date: 28/Jun/19 01:20 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #8904: [BEAM-7389] Add Python snippet for Partition transform URL: https://github.com/apache/beam/pull/8904#discussion_r296033918 ## File path: sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py ## @@ -0,0 +1,87 @@ +# coding=utf-8 +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import absolute_import +from __future__ import print_function + +import unittest + +import mock + +from apache_beam.examples.snippets.transforms.element_wise.partition import * Review comment: Let's make an explicit import instead of a star import. Start imports are not recommended. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268965) Time Spent: 18h (was: 17h 50m) > Colab examples for element-wise transforms (Python) > --- > > Key: BEAM-7389 > URL: https://issues.apache.org/jira/browse/BEAM-7389 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Rose Nguyen >Assignee: David Cavazos >Priority: Minor > Time Spent: 18h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874577#comment-16874577
]
Daniel Oliveira edited comment on BEAM-7648 at 6/28/19 12:37 AM:
-
Actually I may have spoke too soon. The errors seem to be due to beam-15 being
down. I'll leave this bug open for now just in case, but it may be a separate
issue.
*17:02:02* ERROR: apache-beam-jenkins-15 is offline; cannot locate JDK 1.8
(latest)
was (Author: danoliveira):
Actually I may have spoke too soon. The errors seem to be due to beam-15 being
down. I'll leave this bug open for now just in case, but it may be a separate
issue.
*17:02:02* ERROR: apache-beam-jenkins-15 is offline; cannot locate JDK 1.8
(latest)
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
>
[jira] [Updated] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7648:
--
Component/s: test-failures
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: test-failures, testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
> 18:11:49 at com.sun.proxy.$Proxy114.setRemoteUrl(Unknown Source)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
> 18:11:49 at
[jira] [Commented] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874577#comment-16874577
]
Daniel Oliveira commented on BEAM-7648:
---
Actually I may have spoke too soon. The errors seem to be due to beam-15 being
down. I'll leave this bug open for now just in case, but it may be a separate
issue.
*17:02:02* ERROR: apache-beam-jenkins-15 is offline; cannot locate JDK 1.8
(latest)
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
>
[jira] [Updated] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7648:
--
Labels: currently-failing (was: )
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: test-failures, testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Labels: currently-failing
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
> 18:11:49 at com.sun.proxy.$Proxy114.setRemoteUrl(Unknown Source)
> 18:11:49 at
>
[jira] [Reopened] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira reopened BEAM-7648:
---
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
> 18:11:49 at com.sun.proxy.$Proxy114.setRemoteUrl(Unknown Source)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
> 18:11:49 at
[jira] [Commented] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874576#comment-16874576
]
Daniel Oliveira commented on BEAM-7648:
---
Looks like this is back:
[https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Cron/946/]
[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3670/]
(and many others)
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
> 18:11:49 at
[jira] [Work logged] (BEAM-7424) Retry HTTP 429 errors from GCS w/ exponential backoff when reading data
[ https://issues.apache.org/jira/browse/BEAM-7424?focusedWorklogId=268960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268960 ] ASF GitHub Bot logged work on BEAM-7424: Author: ASF GitHub Bot Created on: 28/Jun/19 00:29 Start Date: 28/Jun/19 00:29 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8933: [BEAM-7424] Retry HTTP 429 errors from GCS URL: https://github.com/apache/beam/pull/8933#issuecomment-506557371 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268960) Time Spent: 3h (was: 2h 50m) > Retry HTTP 429 errors from GCS w/ exponential backoff when reading data > --- > > Key: BEAM-7424 > URL: https://issues.apache.org/jira/browse/BEAM-7424 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, io-python-gcp, sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Blocker > Fix For: 2.14.0 > > Time Spent: 3h > Remaining Estimate: 0h > > This has to be done for both Java and Python SDKs. > Seems like Java SDK already retries 429 errors w/o backoff (please verify): > [https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java#L185] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268957=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268957 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 28/Jun/19 00:26 Start Date: 28/Jun/19 00:26 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506548287 Run Python PreCommit -failed because of dataflow issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268957) Time Spent: 18h (was: 17h 50m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 18h > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268949=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268949 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 23:38 Start Date: 27/Jun/19 23:38 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506548287 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268949) Time Spent: 17h 50m (was: 17h 40m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 17h 50m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268948=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268948
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 23:37
Start Date: 27/Jun/19 23:37
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #8959: [BEAM-7548]
Cherry pick - fix flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8959#issuecomment-506548175
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268948)
Time Spent: 5h (was: 4h 50m)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response =
[jira] [Work logged] (BEAM-7424) Retry HTTP 429 errors from GCS w/ exponential backoff when reading data
[ https://issues.apache.org/jira/browse/BEAM-7424?focusedWorklogId=268947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268947 ] ASF GitHub Bot logged work on BEAM-7424: Author: ASF GitHub Bot Created on: 27/Jun/19 23:36 Start Date: 27/Jun/19 23:36 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8933: [BEAM-7424] Retry HTTP 429 errors from GCS URL: https://github.com/apache/beam/pull/8933#issuecomment-506547943 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268947) Time Spent: 2h 50m (was: 2h 40m) > Retry HTTP 429 errors from GCS w/ exponential backoff when reading data > --- > > Key: BEAM-7424 > URL: https://issues.apache.org/jira/browse/BEAM-7424 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, io-python-gcp, sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Blocker > Fix For: 2.14.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > This has to be done for both Java and Python SDKs. > Seems like Java SDK already retries 429 errors w/o backoff (please verify): > [https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java#L185] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7590) Convert PipelineOptionsMap to PipelineOption
[
https://issues.apache.org/jira/browse/BEAM-7590?focusedWorklogId=268943=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268943
]
ASF GitHub Bot logged work on BEAM-7590:
Author: ASF GitHub Bot
Created on: 27/Jun/19 23:27
Start Date: 27/Jun/19 23:27
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #8928: [DO NOT
MERGE] [BEAM-7590] Converting JDBC Pipeline Options Map to PipelineOptions.
URL: https://github.com/apache/beam/pull/8928#discussion_r298405779
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsReflectionSetter.java
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import java.beans.IntrospectionException;
+import java.beans.Introspector;
+import java.beans.PropertyDescriptor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import org.apache.beam.sdk.util.StringUtils;
+import
org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableListMultimap;
+import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import
org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ListMultimap;
+import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+
+/** This is a utility class to set and remove options individually. */
+public class PipelineOptionsReflectionSetter {
+ private static final boolean STRICT_PARSING = true;
+
+ @SuppressWarnings("unchecked")
+ public static Class getPipelineOptionsInterface(
+ PipelineOptions options) {
+if (options.getClass().getInterfaces().length != 1) {
Review comment:
Now that I know your intent, you should instead check to see if the class is
an interface using:
https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html#isInterface--
From `Class#getInterfaces`:
If this object represents an interface, the array contains objects
representing all interfaces extended by the interface.
And hence would fail on things like DataflowPipelineOptions which extends
multiple interfaces.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268943)
Time Spent: 3h 40m (was: 3.5h)
> Convert PipelineOptionsMap to PipelineOption
>
>
> Key: BEAM-7590
> URL: https://issues.apache.org/jira/browse/BEAM-7590
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Alireza Samadianzakaria
>Assignee: Alireza Samadianzakaria
>Priority: Minor
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Currently, BeamCalciteTable keeps a map version of PipelineOptions and that
> map version is used in JDBCConnection and RelNodes as well. This map is empty
> when the pipeline is constructed from SQLTransform and it will have the
> parameters passed from JDBC Client when the pipeline is started by JDBC path.
> Since for Row-Count estimation we need to use PipelineOptions (or its
> sub-classes) and we cannot convert a map that is created from a
> pipelineOptions Subclasses back to PipelineOptions, it is better to keep
> PipelineOptions object itself.
> Another thing that will be changed as a result is set command. Currently, if
> in JDBC we use Set Command for a pipeline option, it will only change that
> option in the map. This means even if the option is incorrect, it does not
> throw exception until it creates the actual Pipeline Options. However, if we
> are keeping the PipelineOptions class itself, then wee need to actually set
> the passed
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268942=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268942 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 23:25 Start Date: 27/Jun/19 23:25 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506535179 Run Python PreCommit - passed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268942) Time Spent: 17h 40m (was: 17.5h) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 17h 40m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7641) collect statistics about python ITs
[ https://issues.apache.org/jira/browse/BEAM-7641?focusedWorklogId=268940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268940 ] ASF GitHub Bot logged work on BEAM-7641: Author: ASF GitHub Bot Created on: 27/Jun/19 23:17 Start Date: 27/Jun/19 23:17 Worklog Time Spent: 10m Work Description: udim commented on issue #8952: [BEAM-7641] Collect xunit statistics for Py ITs URL: https://github.com/apache/beam/pull/8952#issuecomment-506544369 run python postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268940) Time Spent: 3h (was: 2h 50m) > collect statistics about python ITs > --- > > Key: BEAM-7641 > URL: https://issues.apache.org/jira/browse/BEAM-7641 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > Currently ITs don't generate xunit (nosetests.xml) files. > Having this data will make it easier to see which tests failed in a > pre/postcommit run, and to tell if a particular test is flaky. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7641) collect statistics about python ITs
[ https://issues.apache.org/jira/browse/BEAM-7641?focusedWorklogId=268939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268939 ] ASF GitHub Bot logged work on BEAM-7641: Author: ASF GitHub Bot Created on: 27/Jun/19 23:17 Start Date: 27/Jun/19 23:17 Worklog Time Spent: 10m Work Description: udim commented on issue #8952: [BEAM-7641] Collect xunit statistics for Py ITs URL: https://github.com/apache/beam/pull/8952#issuecomment-506544257 run python precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268939) Time Spent: 2h 50m (was: 2h 40m) > collect statistics about python ITs > --- > > Key: BEAM-7641 > URL: https://issues.apache.org/jira/browse/BEAM-7641 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Currently ITs don't generate xunit (nosetests.xml) files. > Having this data will make it easier to see which tests failed in a > pre/postcommit run, and to tell if a particular test is flaky. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
[
https://issues.apache.org/jira/browse/BEAM-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira closed BEAM-7651.
-
Resolution: Duplicate
Fix Version/s: Not applicable
> Failures in various pre and post-commits: "No space left on device"
> ---
>
> Key: BEAM-7651
> URL: https://issues.apache.org/jira/browse/BEAM-7651
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Daniel Oliveira
>Assignee: Mark Liu
>Priority: Major
> Labels: currently-failing
> Fix For: Not applicable
>
>
> I've seen this popping up in several pre and post-commits today. I'll list
> one example with a full message and then several tests where I've seen it
> happen.
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
>
> {noformat}
> 11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
> JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 11:09:46 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 11:09:46 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 11:09:46 at hudson.remoting.Channel.call(Channel.java:955)
> 11:09:46 at hudson.FilePath.act(FilePath.java:1072)
> 11:09:46 at hudson.FilePath.act(FilePath.java:1061)
> 11:09:46 at hudson.FilePath.mkdirs(FilePath.java:1246)
> 11:09:46 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
> 11:09:46 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 11:09:46 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 11:09:46 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 11:09:46 at hudson.model.Run.execute(Run.java:1810)
> 11:09:46 at
> hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 11:09:46 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 11:09:46 at hudson.model.Executor.run(Executor.java:429)
> 11:09:46 java.nio.file.FileSystemException:
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
> No space left on device
> 11:09:46 at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 11:09:46 at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 11:09:46 at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 11:09:46 at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> 11:09:46 at java.nio.file.Files.createDirectory(Files.java:674)
> 11:09:46 at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
> 11:09:46 at java.nio.file.Files.createDirectories(Files.java:767)
> 11:09:46 at hudson.FilePath.mkdirs(FilePath.java:3273)
> 11:09:46 at hudson.FilePath.access$1300(FilePath.java:213)
> 11:09:46 at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
> 11:09:46 at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
> 11:09:46 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
> 11:09:46 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 11:09:46 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 11:09:46 at hudson.remoting.Request$2.run(Request.java:369)
> 11:09:46 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 11:09:46 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 11:09:46 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 11:09:46 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 11:09:46 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 11:09:46 at java.lang.Thread.run(Thread.java:748){noformat}
>
> [https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
> [https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
> [https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
>
> Edit:
> This other failure in several post-commits seems related:
> [https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Cron/1054/]
> {noformat}
> 11:04:49 ERROR: Error fetching remote repo 'origin'
> 11:04:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 11:04:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 11:04:49 at
[jira] [Commented] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
[
https://issues.apache.org/jira/browse/BEAM-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874561#comment-16874561
]
Daniel Oliveira commented on BEAM-7651:
---
Looks like this is a duplicate of
[https://issues.apache.org/jira/browse/BEAM-7648], so I'll close this.
> Failures in various pre and post-commits: "No space left on device"
> ---
>
> Key: BEAM-7651
> URL: https://issues.apache.org/jira/browse/BEAM-7651
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Daniel Oliveira
>Assignee: Mark Liu
>Priority: Major
> Labels: currently-failing
>
> I've seen this popping up in several pre and post-commits today. I'll list
> one example with a full message and then several tests where I've seen it
> happen.
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
>
> {noformat}
> 11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
> JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 11:09:46 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 11:09:46 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 11:09:46 at hudson.remoting.Channel.call(Channel.java:955)
> 11:09:46 at hudson.FilePath.act(FilePath.java:1072)
> 11:09:46 at hudson.FilePath.act(FilePath.java:1061)
> 11:09:46 at hudson.FilePath.mkdirs(FilePath.java:1246)
> 11:09:46 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
> 11:09:46 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 11:09:46 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 11:09:46 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 11:09:46 at hudson.model.Run.execute(Run.java:1810)
> 11:09:46 at
> hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 11:09:46 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 11:09:46 at hudson.model.Executor.run(Executor.java:429)
> 11:09:46 java.nio.file.FileSystemException:
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
> No space left on device
> 11:09:46 at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 11:09:46 at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 11:09:46 at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 11:09:46 at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> 11:09:46 at java.nio.file.Files.createDirectory(Files.java:674)
> 11:09:46 at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
> 11:09:46 at java.nio.file.Files.createDirectories(Files.java:767)
> 11:09:46 at hudson.FilePath.mkdirs(FilePath.java:3273)
> 11:09:46 at hudson.FilePath.access$1300(FilePath.java:213)
> 11:09:46 at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
> 11:09:46 at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
> 11:09:46 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
> 11:09:46 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 11:09:46 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 11:09:46 at hudson.remoting.Request$2.run(Request.java:369)
> 11:09:46 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 11:09:46 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 11:09:46 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 11:09:46 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 11:09:46 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 11:09:46 at java.lang.Thread.run(Thread.java:748){noformat}
>
> [https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
> [https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
> [https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
>
> Edit:
> This other failure in several post-commits seems related:
> [https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Cron/1054/]
> {noformat}
> 11:04:49 ERROR: Error fetching remote repo 'origin'
> 11:04:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 11:04:49 at
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268930=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268930 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 22:46 Start Date: 27/Jun/19 22:46 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506501896 Run Python PreCommit - failed. ERROR: test_split_half_sdf (apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTestWithMultiWorkers) PY2Gcp Link: https://scans.gradle.com/s/ub6eaxz7pqpxs/console-log?task=:sdks:python:testPy2Gcp#L6205 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268930) Time Spent: 17.5h (was: 17h 20m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 17.5h > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7545) Row Count Estimation for CSV TextTable
[ https://issues.apache.org/jira/browse/BEAM-7545?focusedWorklogId=268929=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268929 ] ASF GitHub Bot logged work on BEAM-7545: Author: ASF GitHub Bot Created on: 27/Jun/19 22:33 Start Date: 27/Jun/19 22:33 Worklog Time Spent: 10m Work Description: riazela commented on issue #8951: [BEAM-7545] Adding RowCount to TextTable URL: https://github.com/apache/beam/pull/8951#issuecomment-506535328 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268929) Time Spent: 3h 50m (was: 3h 40m) > Row Count Estimation for CSV TextTable > -- > > Key: BEAM-7545 > URL: https://issues.apache.org/jira/browse/BEAM-7545 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Alireza Samadianzakaria >Assignee: Alireza Samadianzakaria >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > Implementing Row Count Estimation for CSV Tables by reading the first few > lines of the file and estimating the number of records based on the length of > these lines and the total length of the file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7545) Row Count Estimation for CSV TextTable
[ https://issues.apache.org/jira/browse/BEAM-7545?focusedWorklogId=268928=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268928 ] ASF GitHub Bot logged work on BEAM-7545: Author: ASF GitHub Bot Created on: 27/Jun/19 22:33 Start Date: 27/Jun/19 22:33 Worklog Time Spent: 10m Work Description: riazela commented on issue #8951: [BEAM-7545] Adding RowCount to TextTable URL: https://github.com/apache/beam/pull/8951#issuecomment-506535284 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268928) Time Spent: 3h 40m (was: 3.5h) > Row Count Estimation for CSV TextTable > -- > > Key: BEAM-7545 > URL: https://issues.apache.org/jira/browse/BEAM-7545 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Alireza Samadianzakaria >Assignee: Alireza Samadianzakaria >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > Implementing Row Count Estimation for CSV Tables by reading the first few > lines of the file and estimating the number of records based on the length of > these lines and the total length of the file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268925 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 22:33 Start Date: 27/Jun/19 22:33 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506534758 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268925) Time Spent: 17h 10m (was: 17h) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 17h 10m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268926=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268926 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 22:33 Start Date: 27/Jun/19 22:33 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506535179 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268926) Time Spent: 17h 20m (was: 17h 10m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 17h 20m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268924=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268924 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 22:33 Start Date: 27/Jun/19 22:33 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506457502 Run Python PreCommit - passed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268924) Time Spent: 17h (was: 16h 50m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 17h > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268922=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268922 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 22:32 Start Date: 27/Jun/19 22:32 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506534758 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268922) Time Spent: 16h 40m (was: 16.5h) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 16h 40m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268921=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268921 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 22:32 Start Date: 27/Jun/19 22:32 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506534758 Run Python PreCommit - failed. ERROR: test_split_half_sdf (apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTestWithMultiWorkers) PY2Gcp This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268921) Time Spent: 16.5h (was: 16h 20m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 16.5h > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268923=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268923 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 22:32 Start Date: 27/Jun/19 22:32 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506501896 Run Python PreCommit - failed. ERROR: test_split_half_sdf (apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTestWithMultiWorkers) PY2Gcp This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268923) Time Spent: 16h 50m (was: 16h 40m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 16h 50m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268918=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268918
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 22:31
Start Date: 27/Jun/19 22:31
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #8959: [BEAM-7548]
Cherry pick - fix flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8959#issuecomment-506534721
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268918)
Time Spent: 4h 50m (was: 4h 40m)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response =
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268919=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268919 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 22:31 Start Date: 27/Jun/19 22:31 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506534758 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268919) Time Spent: 16h 10m (was: 16h) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 16h 10m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268920=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268920 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 22:31 Start Date: 27/Jun/19 22:31 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506501896 Run Python PreCommit - passed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268920) Time Spent: 16h 20m (was: 16h 10m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 16h 20m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874544#comment-16874544
]
yifan zou commented on BEAM-7648:
-
Stale docker images fill up the disk. Removed the image and the worker is back
online.
Opened the https://issues.apache.org/jira/browse/BEAM-7650 to track the docker
issue.
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
> 18:11:49 at com.sun.proxy.$Proxy114.setRemoteUrl(Unknown Source)
>
[jira] [Closed] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yifan zou closed BEAM-7648.
---
Resolution: Fixed
Fix Version/s: 2.15.0
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
> 18:11:49 at com.sun.proxy.$Proxy114.setRemoteUrl(Unknown Source)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
> 18:11:49 at
[jira] [Work logged] (BEAM-7586) Add Integration Test for MongoDbIO in python sdk
[ https://issues.apache.org/jira/browse/BEAM-7586?focusedWorklogId=268900=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268900 ] ASF GitHub Bot logged work on BEAM-7586: Author: ASF GitHub Bot Created on: 27/Jun/19 22:16 Start Date: 27/Jun/19 22:16 Worklog Time Spent: 10m Work Description: y1chi commented on issue #8895: [BEAM-7586] Add Integration test for python mongodb io URL: https://github.com/apache/beam/pull/8895#issuecomment-506531126 @chamikaramj do you mind take another look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268900) Time Spent: 8.5h (was: 8h 20m) > Add Integration Test for MongoDbIO in python sdk > - > > Key: BEAM-7586 > URL: https://issues.apache.org/jira/browse/BEAM-7586 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7650) Python Portable and HDFS IT need to clean up the docker image after complete
[ https://issues.apache.org/jira/browse/BEAM-7650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yifan zou updated BEAM-7650: Description: The python portable test and HDFS IT (a part of postcommit?) generated docker images but does not clean them properly. This eventually run out of disk of the build worker. REPOSITORY TAG IMAGE ID CREATED SIZE SHARED SIZE UNIQUE SIZE CONTAINERS jenkins-docker-apache.bintray.io/beam/python3 latest ff1b949f4442 22 hours ago 1.639GB 922.3MB 716.9MB 0 jenkins-docker-apache.bintray.io/beam/python latest 1dda7b9d9748 22 hours ago 1.624GB 913.7MB 710.3MB 0 05458187a0e3 22 hours ago 732.9MB 625.1MB 107.8MB 4 896f35dd685f 23 hours ago 1.639GB 922.3MB 716.9MB 0 db4d24ca9f2b 23 hours ago 1.624GB 913.7MB 710.3MB 0 547df4d71c31 23 hours ago 732.9MB 625.1MB 107.8MB 4 dd7d9582c3e0 23 hours ago 1.639GB 922.3MB 716.9MB 0 664aae255239 23 hours ago 1.624GB 913.7MB 710.3MB 0 b528fedf9228 23 hours ago 732.9MB 625.1MB 107.8MB 4 8e996f22435e 25 hours ago 1.624GB 913.7MB 710.3MB 0 hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_test latest 24b73b3fec06 25 hours ago 1.305GB 965.7MB 339.5MB 0 096325fb48de 25 hours ago 732.9MB 625.1MB 107.8MB 2 jenkins-docker-apache.bintray.io/beam/java latest c36d8ff2945d 25 hours ago 685.6MB 625.1MB 60.52MB 0 11c86ebe025f 26 hours ago 1.639GB 922.3MB 716.9MB 0 2ecd69c89ec1 26 hours ago 1.624GB 913.7MB 710.3MB 0 hdfs_it-jenkins-beam_postcommit_python_verify-8590_test latest 3d1d589d44fe 2 days ago 1.305GB 965.7MB 339.5MB 0* * hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test latest d1cc503ebe8e 2 days ago 1.305GB 965.7MB 339.2MB 0* * hdfs_it-jenkins-beam_postcommit_python_verify-8577_test latest 8582c6ca6e15 3 days ago 1.305GB 965.7MB 339.2MB 0* * hdfs_it-jenkins-beam_postcommit_python_verify-8576_test latest 4591e0948170 3 days ago 1.305GB 965.7MB 339.2MB 0* * hdfs_it-jenkins-beam_postcommit_python_verify-8575_test latest ab181c49d56e 4 days ago 1.305GB 965.7MB 339.2MB 0* * hdfs_it-jenkins-beam_postcommit_python_verify-8573_test latest 2104ba0a6db7 4 days ago 1.305GB 965.7MB 339.2MB 0* * ... <1000+ images> was:The python portable test and HDFS IT (a part of postcommit?) generated docker images but does not clean them properly. This eventually run out of disk of the build worker. >
[jira] [Updated] (BEAM-7650) Python Portable and HDFS IT need to clean up the docker image after complete
[ https://issues.apache.org/jira/browse/BEAM-7650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yifan zou updated BEAM-7650: Description: The python portable test and HDFS IT (a part of postcommit?) generated docker images but does not clean them properly. This eventually run out of disk of the build worker. (was: The python portable test generated docker images but does not clean them properly. This eventually run out of disk of the build worker.) > Python Portable and HDFS IT need to clean up the docker image after complete > > > Key: BEAM-7650 > URL: https://issues.apache.org/jira/browse/BEAM-7650 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: yifan zou >Assignee: Ankur Goenka >Priority: Critical > > The python portable test and HDFS IT (a part of postcommit?) generated docker > images but does not clean them properly. This eventually run out of disk of > the build worker. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7650) Python Portable and HDFS IT need to clean up the docker image after complete
[ https://issues.apache.org/jira/browse/BEAM-7650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yifan zou updated BEAM-7650: Summary: Python Portable and HDFS IT need to clean up the docker image after complete (was: Python Portable test needs to clean up the docker image after complete) > Python Portable and HDFS IT need to clean up the docker image after complete > > > Key: BEAM-7650 > URL: https://issues.apache.org/jira/browse/BEAM-7650 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: yifan zou >Assignee: Ankur Goenka >Priority: Critical > > The python portable test generated docker images but does not clean them > properly. This eventually run out of disk of the build worker. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7650) Python Portable test needs to clean up the docker image after complete
[ https://issues.apache.org/jira/browse/BEAM-7650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yifan zou updated BEAM-7650: Description: The python portable test generated docker images but does not clean them properly. This eventually run out of disk of the build worker. > Python Portable test needs to clean up the docker image after complete > -- > > Key: BEAM-7650 > URL: https://issues.apache.org/jira/browse/BEAM-7650 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: yifan zou >Assignee: Ankur Goenka >Priority: Critical > > The python portable test generated docker images but does not clean them > properly. This eventually run out of disk of the build worker. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
[
https://issues.apache.org/jira/browse/BEAM-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7651:
--
Description:
I've seen this popping up in several pre and post-commits today. I'll list one
example with a full message and then several tests where I've seen it happen.
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
{noformat}
11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
JNLP4-connect connection from
103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
11:09:46at
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
11:09:46at
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
11:09:46at hudson.remoting.Channel.call(Channel.java:955)
11:09:46at hudson.FilePath.act(FilePath.java:1072)
11:09:46at hudson.FilePath.act(FilePath.java:1061)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:1246)
11:09:46at
hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
11:09:46at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
11:09:46at hudson.model.Run.execute(Run.java:1810)
11:09:46at
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
11:09:46at
hudson.model.ResourceController.execute(ResourceController.java:97)
11:09:46at hudson.model.Executor.run(Executor.java:429)
11:09:46 java.nio.file.FileSystemException:
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
No space left on device
11:09:46at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
11:09:46at
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
11:09:46at java.nio.file.Files.createDirectory(Files.java:674)
11:09:46at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
11:09:46at java.nio.file.Files.createDirectories(Files.java:767)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:3273)
11:09:46at hudson.FilePath.access$1300(FilePath.java:213)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
11:09:46at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:212)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:54)
11:09:46at hudson.remoting.Request$2.run(Request.java:369)
11:09:46at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
11:09:46at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11:09:46at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
11:09:46at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
11:09:46at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
11:09:46at java.lang.Thread.run(Thread.java:748){noformat}
[https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
[https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
[https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
Edit:
This other failure in several post-commits seems related:
[https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Cron/1054/]
{noformat}
11:04:49 ERROR: Error fetching remote repo 'origin'
11:04:49 hudson.plugins.git.GitException: Failed to fetch from
https://github.com/apache/beam.git
11:04:49at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
11:04:49at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
11:04:49at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
11:04:49at hudson.scm.SCM.checkout(SCM.java:504)
11:04:49at
hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
11:04:49at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
11:04:49at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:04:49at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
11:04:49at hudson.model.Run.execute(Run.java:1810)
11:04:49at
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268868=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268868
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 21:48
Start Date: 27/Jun/19 21:48
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #8959: [BEAM-7548]
Cherry pick - fix flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8959#issuecomment-506523972
R: @akedin
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268868)
Time Spent: 4h 40m (was: 4.5h)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response =
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[ https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268867=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268867 ] ASF GitHub Bot logged work on BEAM-7548: Author: ASF GitHub Bot Created on: 27/Jun/19 21:45 Start Date: 27/Jun/19 21:45 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #8959: [BEAM-7548] Cherry pick - fix flaky tests for ApproximateUnique URL: https://github.com/apache/beam/pull/8959 Flaky tests for ApproximateUnique are fixed. Now cherry picking it to release-2.14.0 branch. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [![Build
[jira] [Updated] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
[
https://issues.apache.org/jira/browse/BEAM-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7651:
--
Description:
I've seen this popping up in several pre and post-commits today. I'll list one
example with a full message and then several tests where I've seen it happen.
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
{noformat}
11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
JNLP4-connect connection from
103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
11:09:46at
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
11:09:46at
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
11:09:46at hudson.remoting.Channel.call(Channel.java:955)
11:09:46at hudson.FilePath.act(FilePath.java:1072)
11:09:46at hudson.FilePath.act(FilePath.java:1061)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:1246)
11:09:46at
hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
11:09:46at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
11:09:46at hudson.model.Run.execute(Run.java:1810)
11:09:46at
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
11:09:46at
hudson.model.ResourceController.execute(ResourceController.java:97)
11:09:46at hudson.model.Executor.run(Executor.java:429)
11:09:46 java.nio.file.FileSystemException:
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
No space left on device
11:09:46at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
11:09:46at
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
11:09:46at java.nio.file.Files.createDirectory(Files.java:674)
11:09:46at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
11:09:46at java.nio.file.Files.createDirectories(Files.java:767)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:3273)
11:09:46at hudson.FilePath.access$1300(FilePath.java:213)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
11:09:46at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:212)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:54)
11:09:46at hudson.remoting.Request$2.run(Request.java:369)
11:09:46at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
11:09:46at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11:09:46at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
11:09:46at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
11:09:46at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
11:09:46at java.lang.Thread.run(Thread.java:748){noformat}
[https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
[https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
[https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
Edit:
This other failure in several post-commits seems related:
[https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Cron/1054/]
{noformat}
11:04:49 ERROR: Error fetching remote repo 'origin'
11:04:49 hudson.plugins.git.GitException: Failed to fetch from
https://github.com/apache/beam.git
11:04:49at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
11:04:49at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
11:04:49at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
11:04:49at hudson.scm.SCM.checkout(SCM.java:504)
11:04:49at
hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
11:04:49at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
11:04:49at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:04:49at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
11:04:49at hudson.model.Run.execute(Run.java:1810)
11:04:49at
[jira] [Assigned] (BEAM-7650) Python Portable test needs to clean up the docker image after complete
[ https://issues.apache.org/jira/browse/BEAM-7650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yifan zou reassigned BEAM-7650: --- Assignee: Ankur Goenka (was: Luke Cwik) > Python Portable test needs to clean up the docker image after complete > -- > > Key: BEAM-7650 > URL: https://issues.apache.org/jira/browse/BEAM-7650 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: yifan zou >Assignee: Ankur Goenka >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-7650) Python Portable test needs to clean up the docker image after complete
[ https://issues.apache.org/jira/browse/BEAM-7650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yifan zou reassigned BEAM-7650: --- Assignee: Luke Cwik > Python Portable test needs to clean up the docker image after complete > -- > > Key: BEAM-7650 > URL: https://issues.apache.org/jira/browse/BEAM-7650 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: yifan zou >Assignee: Luke Cwik >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
[
https://issues.apache.org/jira/browse/BEAM-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874520#comment-16874520
]
Daniel Oliveira commented on BEAM-7651:
---
Mark, this sounds like some sort of quota issue maybe. Could you take look?
> Failures in various pre and post-commits: "No space left on device"
> ---
>
> Key: BEAM-7651
> URL: https://issues.apache.org/jira/browse/BEAM-7651
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Daniel Oliveira
>Assignee: Mark Liu
>Priority: Major
> Labels: currently-failing
>
> I've seen this popping up in several pre and post-commits today. I'll list
> one example with a full message and then several tests where I've seen it
> happen.
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
>
> {noformat}
> 11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
> JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 11:09:46 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 11:09:46 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 11:09:46 at hudson.remoting.Channel.call(Channel.java:955)
> 11:09:46 at hudson.FilePath.act(FilePath.java:1072)
> 11:09:46 at hudson.FilePath.act(FilePath.java:1061)
> 11:09:46 at hudson.FilePath.mkdirs(FilePath.java:1246)
> 11:09:46 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
> 11:09:46 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 11:09:46 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 11:09:46 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 11:09:46 at hudson.model.Run.execute(Run.java:1810)
> 11:09:46 at
> hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 11:09:46 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 11:09:46 at hudson.model.Executor.run(Executor.java:429)
> 11:09:46 java.nio.file.FileSystemException:
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
> No space left on device
> 11:09:46 at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 11:09:46 at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 11:09:46 at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 11:09:46 at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> 11:09:46 at java.nio.file.Files.createDirectory(Files.java:674)
> 11:09:46 at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
> 11:09:46 at java.nio.file.Files.createDirectories(Files.java:767)
> 11:09:46 at hudson.FilePath.mkdirs(FilePath.java:3273)
> 11:09:46 at hudson.FilePath.access$1300(FilePath.java:213)
> 11:09:46 at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
> 11:09:46 at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
> 11:09:46 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
> 11:09:46 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 11:09:46 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 11:09:46 at hudson.remoting.Request$2.run(Request.java:369)
> 11:09:46 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 11:09:46 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 11:09:46 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 11:09:46 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 11:09:46 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 11:09:46 at java.lang.Thread.run(Thread.java:748){noformat}
>
> [https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
> [https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
> [https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
[
https://issues.apache.org/jira/browse/BEAM-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7651:
--
Description:
I've seen this popping up in several pre and post-commits today. I'll list one
example with a full message and then several tests where I've seen it happen.
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
{noformat}
11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
JNLP4-connect connection from
103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
11:09:46at
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
11:09:46at
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
11:09:46at hudson.remoting.Channel.call(Channel.java:955)
11:09:46at hudson.FilePath.act(FilePath.java:1072)
11:09:46at hudson.FilePath.act(FilePath.java:1061)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:1246)
11:09:46at
hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
11:09:46at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
11:09:46at hudson.model.Run.execute(Run.java:1810)
11:09:46at
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
11:09:46at
hudson.model.ResourceController.execute(ResourceController.java:97)
11:09:46at hudson.model.Executor.run(Executor.java:429)
11:09:46 java.nio.file.FileSystemException:
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
No space left on device
11:09:46at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
11:09:46at
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
11:09:46at java.nio.file.Files.createDirectory(Files.java:674)
11:09:46at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
11:09:46at java.nio.file.Files.createDirectories(Files.java:767)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:3273)
11:09:46at hudson.FilePath.access$1300(FilePath.java:213)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
11:09:46at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:212)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:54)
11:09:46at hudson.remoting.Request$2.run(Request.java:369)
11:09:46at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
11:09:46at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11:09:46at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
11:09:46at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
11:09:46at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
11:09:46at java.lang.Thread.run(Thread.java:748){noformat}
[https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
[https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
[https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
was:
I've seen this popping up in several pre and post-commits today. I'll list one
example with a full message and then several tests where I've seen it happen.
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
{noformat}
11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
JNLP4-connect connection from
103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
11:09:46at
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
11:09:46at
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
11:09:46at hudson.remoting.Channel.call(Channel.java:955)
11:09:46at hudson.FilePath.act(FilePath.java:1072)
11:09:46at hudson.FilePath.act(FilePath.java:1061)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:1246)
11:09:46at
hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
11:09:46at
[jira] [Created] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
Daniel Oliveira created BEAM-7651:
-
Summary: Failures in various pre and post-commits: "No space left
on device"
Key: BEAM-7651
URL: https://issues.apache.org/jira/browse/BEAM-7651
Project: Beam
Issue Type: Bug
Components: test-failures
Reporter: Daniel Oliveira
Assignee: Mark Liu
I've seen this popping up in several pre and post-commits today. I'll list one
example with a full message and then several tests where I've seen it happen.
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
{noformat}
11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
JNLP4-connect connection from
103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
11:09:46at
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
11:09:46at
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
11:09:46at hudson.remoting.Channel.call(Channel.java:955)
11:09:46at hudson.FilePath.act(FilePath.java:1072)
11:09:46at hudson.FilePath.act(FilePath.java:1061)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:1246)
11:09:46at
hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
11:09:46at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
11:09:46at hudson.model.Run.execute(Run.java:1810)
11:09:46at
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
11:09:46at
hudson.model.ResourceController.execute(ResourceController.java:97)
11:09:46at hudson.model.Executor.run(Executor.java:429)
11:09:46 java.nio.file.FileSystemException:
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
No space left on device
11:09:46at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
11:09:46at
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
11:09:46at java.nio.file.Files.createDirectory(Files.java:674)
11:09:46at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
11:09:46at java.nio.file.Files.createDirectories(Files.java:767)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:3273)
11:09:46at hudson.FilePath.access$1300(FilePath.java:213)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
11:09:46at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:212)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:54)
11:09:46at hudson.remoting.Request$2.run(Request.java:369)
11:09:46at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
11:09:46at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11:09:46at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
11:09:46at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
11:09:46at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
11:09:46at java.lang.Thread.run(Thread.java:748){noformat}
[https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
[https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
[https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-7650) Python Portable test needs to clean up the docker image after complete
yifan zou created BEAM-7650: --- Summary: Python Portable test needs to clean up the docker image after complete Key: BEAM-7650 URL: https://issues.apache.org/jira/browse/BEAM-7650 Project: Beam Issue Type: Bug Components: testing Reporter: yifan zou -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5795) Can SQL Query 5 be simplified?
[ https://issues.apache.org/jira/browse/BEAM-5795?focusedWorklogId=268839=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268839 ] ASF GitHub Bot logged work on BEAM-5795: Author: ASF GitHub Bot Created on: 27/Jun/19 21:18 Start Date: 27/Jun/19 21:18 Worklog Time Spent: 10m Work Description: stale[bot] commented on issue #6757: [BEAM-5795] Simplify SQL Query 5 URL: https://github.com/apache/beam/pull/6757#issuecomment-506515507 This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268839) Time Spent: 1.5h (was: 1h 20m) > Can SQL Query 5 be simplified? > -- > > Key: BEAM-5795 > URL: https://issues.apache.org/jira/browse/BEAM-5795 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > The original CQL query uses the ALL operator over the set of rows that are > within a certain period from the watermark. We instead have a fancy join, due > to windowing. Nonetheless, can this be simplified? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5795) Can SQL Query 5 be simplified?
[ https://issues.apache.org/jira/browse/BEAM-5795?focusedWorklogId=268840=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268840 ] ASF GitHub Bot logged work on BEAM-5795: Author: ASF GitHub Bot Created on: 27/Jun/19 21:18 Start Date: 27/Jun/19 21:18 Worklog Time Spent: 10m Work Description: stale[bot] commented on pull request #6757: [BEAM-5795] Simplify SQL Query 5 URL: https://github.com/apache/beam/pull/6757 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268840) Time Spent: 1h 40m (was: 1.5h) > Can SQL Query 5 be simplified? > -- > > Key: BEAM-5795 > URL: https://issues.apache.org/jira/browse/BEAM-5795 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > The original CQL query uses the ALL operator over the set of rows that are > within a certain period from the watermark. We instead have a fancy join, due > to windowing. Nonetheless, can this be simplified? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7545) Row Count Estimation for CSV TextTable
[
https://issues.apache.org/jira/browse/BEAM-7545?focusedWorklogId=268832=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268832
]
ASF GitHub Bot logged work on BEAM-7545:
Author: ASF GitHub Bot
Created on: 27/Jun/19 21:04
Start Date: 27/Jun/19 21:04
Worklog Time Spent: 10m
Work Description: riazela commented on pull request #8951: [BEAM-7545]
Adding RowCount to TextTable
URL: https://github.com/apache/beam/pull/8951#discussion_r298369731
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextRowCountEstimator.java
##
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io;
+
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.io.fs.EmptyMatchTreatment;
+import org.apache.beam.sdk.io.fs.MatchResult;
+import org.apache.beam.sdk.io.range.OffsetRange;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.ValueProvider;
+
+/** This returns a row count estimation for files associated with a file
pattern. */
+@AutoValue
+public abstract class TextRowCountEstimator {
+ private static final long DEFAULT_NUM_BYTES_LINES = 1024L;
+ private static final Compression DEFAULT_COMPRESSION = Compression.AUTO;
+ private static final FileIO.ReadMatches.DirectoryTreatment
DEFAULT_DIRECTORY_TREATMENT =
+ FileIO.ReadMatches.DirectoryTreatment.SKIP;
+ private static final EmptyMatchTreatment DEFAULT_EMPTY_MATCH_TREATMENT =
+ EmptyMatchTreatment.DISALLOW;
+
+ public abstract int getNumSampledFiles();
+
+ public boolean isSamplingAllFiles() {
+return getNumSampledFiles() == -1;
+ }
+
+ public abstract long getNumSampledBytes();
+
+ @Nullable
+ @SuppressWarnings("mutable")
+ public abstract byte[] getDelimiters();
+
+ public abstract String getFilePattern();
+
+ public abstract Compression getCompression();
+
+ public abstract EmptyMatchTreatment getEmptyMatchTreatment();
+
+ public abstract FileIO.ReadMatches.DirectoryTreatment
getDirectoryTreatment();
+
+ public static TextRowCountEstimator.Builder builder() {
+return (new AutoValue_TextRowCountEstimator.Builder())
+.sampleAllFiles()
+.setNumSampledBytes(DEFAULT_NUM_BYTES_LINES)
+.setCompression(DEFAULT_COMPRESSION)
+.setDirectoryTreatment(DEFAULT_DIRECTORY_TREATMENT)
+.setEmptyMatchTreatment(DEFAULT_EMPTY_MATCH_TREATMENT);
+ }
+
+ /** Builder for {@link org.apache.beam.sdk.io.TextRowCountEstimator}. */
+ @AutoValue.Builder
+ public abstract static class Builder {
+
+public abstract Builder setNumSampledFiles(int numSampledFiles);
+
+public abstract Builder setNumSampledBytes(long numSampledBytes);
+
+public abstract Builder setDirectoryTreatment(
+FileIO.ReadMatches.DirectoryTreatment directoryTreatment);
+
+public abstract Builder setCompression(Compression compression);
+
+public abstract Builder setDelimiters(byte[] delimiters);
+
+public abstract Builder setFilePattern(String filePattern);
+
+public abstract Builder setEmptyMatchTreatment(EmptyMatchTreatment
emptyMatchTreatment);
+
+public Builder sampleAllFiles() {
+ return this.setNumSampledFiles(-1);
+}
+
+public abstract TextRowCountEstimator build();
+ }
+
+ /**
+ * Estimates the number of non empty rows. Note that it does not cache the
previous estimate. It
+ * returns 0L if all the lines that have been read are empty.
+ *
+ * @return Number of estimated rows.
+ * @throws IOException
+ */
+ public Long estimateRowCount(PipelineOptions pipelineOptions) throws
IOException {
+long linesSize = 0;
+int numberOfReadLines = 0;
+long totalFileSizes = 0;
+int numberOfReadFiles = 0;
+
+MatchResult match = FileSystems.match(getFilePattern(),
getEmptyMatchTreatment());
+
+for (MatchResult.Metadata metadata : match.metadata()) {
+
+ if ((!isSamplingAllFiles()) && numberOfReadFiles > getNumSampledFiles())
{
+break;
+ }
+
+ FileIO.ReadableFile file =
+
[jira] [Work logged] (BEAM-7603) Support for ValueProvider-given GCS Location for WriteToBigQuery w File Loads
[ https://issues.apache.org/jira/browse/BEAM-7603?focusedWorklogId=268819=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268819 ] ASF GitHub Bot logged work on BEAM-7603: Author: ASF GitHub Bot Created on: 27/Jun/19 20:53 Start Date: 27/Jun/19 20:53 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8908: [BEAM-7603] Support for ValueProvider-given GCS Location for WriteToBigQuery w File Loads URL: https://github.com/apache/beam/pull/8908#issuecomment-506507230 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268819) Time Spent: 3h 50m (was: 3h 40m) > Support for ValueProvider-given GCS Location for WriteToBigQuery w File Loads > - > > Key: BEAM-7603 > URL: https://issues.apache.org/jira/browse/BEAM-7603 > Project: Beam > Issue Type: Improvement > Components: io-python-gcp >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Blocker > Fix For: 2.14.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7424) Retry HTTP 429 errors from GCS w/ exponential backoff when reading data
[ https://issues.apache.org/jira/browse/BEAM-7424?focusedWorklogId=268816=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268816 ] ASF GitHub Bot logged work on BEAM-7424: Author: ASF GitHub Bot Created on: 27/Jun/19 20:52 Start Date: 27/Jun/19 20:52 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8933: [BEAM-7424] Retry HTTP 429 errors from GCS URL: https://github.com/apache/beam/pull/8933#issuecomment-506506868 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268816) Time Spent: 2h 40m (was: 2.5h) > Retry HTTP 429 errors from GCS w/ exponential backoff when reading data > --- > > Key: BEAM-7424 > URL: https://issues.apache.org/jira/browse/BEAM-7424 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, io-python-gcp, sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Blocker > Fix For: 2.14.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > This has to be done for both Java and Python SDKs. > Seems like Java SDK already retries 429 errors w/o backoff (please verify): > [https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java#L185] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7586) Add Integration Test for MongoDbIO in python sdk
[ https://issues.apache.org/jira/browse/BEAM-7586?focusedWorklogId=268811=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268811 ] ASF GitHub Bot logged work on BEAM-7586: Author: ASF GitHub Bot Created on: 27/Jun/19 20:42 Start Date: 27/Jun/19 20:42 Worklog Time Spent: 10m Work Description: y1chi commented on issue #8895: [BEAM-7586] Add Integration test for python mongodb io URL: https://github.com/apache/beam/pull/8895#issuecomment-506425101 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268811) Time Spent: 8h 20m (was: 8h 10m) > Add Integration Test for MongoDbIO in python sdk > - > > Key: BEAM-7586 > URL: https://issues.apache.org/jira/browse/BEAM-7586 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268808 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 20:38 Start Date: 27/Jun/19 20:38 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506501896 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268808) Time Spent: 16h (was: 15h 50m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 16h > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268806=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268806
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 20:37
Start Date: 27/Jun/19 20:37
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #8948: [BEAM-7548] fix
flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8948#issuecomment-506501646
> LGTM, thank you!
Thank you @tvalentyn
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268806)
Time Spent: 4h 10m (was: 4h)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268807=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268807
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 20:37
Start Date: 27/Jun/19 20:37
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #8948:
[BEAM-7548] fix flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8948
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268807)
Time Spent: 4h 20m (was: 4h 10m)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response = self.worker.do_instruction(request)
> 11:57:47 File
[jira] [Work logged] (BEAM-7586) Add Integration Test for MongoDbIO in python sdk
[ https://issues.apache.org/jira/browse/BEAM-7586?focusedWorklogId=268797=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268797 ] ASF GitHub Bot logged work on BEAM-7586: Author: ASF GitHub Bot Created on: 27/Jun/19 20:12 Start Date: 27/Jun/19 20:12 Worklog Time Spent: 10m Work Description: y1chi commented on issue #8895: [BEAM-7586] Add Integration test for python mongodb io URL: https://github.com/apache/beam/pull/8895#issuecomment-506493130 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268797) Time Spent: 8h 10m (was: 8h) > Add Integration Test for MongoDbIO in python sdk > - > > Key: BEAM-7586 > URL: https://issues.apache.org/jira/browse/BEAM-7586 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 8h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7586) Add Integration Test for MongoDbIO in python sdk
[ https://issues.apache.org/jira/browse/BEAM-7586?focusedWorklogId=268796=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268796 ] ASF GitHub Bot logged work on BEAM-7586: Author: ASF GitHub Bot Created on: 27/Jun/19 20:12 Start Date: 27/Jun/19 20:12 Worklog Time Spent: 10m Work Description: y1chi commented on issue #8895: [BEAM-7586] Add Integration test for python mongodb io URL: https://github.com/apache/beam/pull/8895#issuecomment-506447539 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268796) Time Spent: 8h (was: 7h 50m) > Add Integration Test for MongoDbIO in python sdk > - > > Key: BEAM-7586 > URL: https://issues.apache.org/jira/browse/BEAM-7586 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268787=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268787 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 19:51 Start Date: 27/Jun/19 19:51 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506486376 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268787) Time Spent: 15h 50m (was: 15h 40m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 15h 50m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268783=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268783 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 19:38 Start Date: 27/Jun/19 19:38 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506482566 Run Python_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268783) Time Spent: 15h 40m (was: 15.5h) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 15h 40m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268781 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 19:38 Start Date: 27/Jun/19 19:38 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506479835 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268781) Time Spent: 15h 20m (was: 15h 10m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 15h 20m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268782=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268782 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 19:38 Start Date: 27/Jun/19 19:38 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506482498 Run RAT PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268782) Time Spent: 15.5h (was: 15h 20m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 15.5h > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268770=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268770 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 19:29 Start Date: 27/Jun/19 19:29 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506479835 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268770) Time Spent: 15h 10m (was: 15h) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 15h 10m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268769=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268769 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 19:29 Start Date: 27/Jun/19 19:29 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506479835 Run Portable_Python PreCommit Run Python_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268769) Time Spent: 15h (was: 14h 50m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 15h > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268768
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 19:28
Start Date: 27/Jun/19 19:28
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #8948: [BEAM-7548] fix
flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8948#issuecomment-506479504
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268768)
Time Spent: 4h (was: 3h 50m)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response =
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268741
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 18:55
Start Date: 27/Jun/19 18:55
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #8948: [BEAM-7548] fix
flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8948#issuecomment-506469137
Run Python_PVR_Flink PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268741)
Time Spent: 3h 50m (was: 3h 40m)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response =
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268719=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268719
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 18:28
Start Date: 27/Jun/19 18:28
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #8948: [BEAM-7548] fix
flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8948#issuecomment-506459215
Run Portable_Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268719)
Time Spent: 3h 40m (was: 3.5h)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response =
[jira] [Work logged] (BEAM-3645) Support multi-process execution on the FnApiRunner
[ https://issues.apache.org/jira/browse/BEAM-3645?focusedWorklogId=268718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268718 ] ASF GitHub Bot logged work on BEAM-3645: Author: ASF GitHub Bot Created on: 27/Jun/19 18:23 Start Date: 27/Jun/19 18:23 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #8872: [BEAM-3645] add ParallelBundleManager URL: https://github.com/apache/beam/pull/8872#issuecomment-506457502 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268718) Time Spent: 14h 50m (was: 14h 40m) > Support multi-process execution on the FnApiRunner > -- > > Key: BEAM-3645 > URL: https://issues.apache.org/jira/browse/BEAM-3645 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.2.0, 2.3.0 >Reporter: Charles Chen >Assignee: Hannah Jiang >Priority: Major > Time Spent: 14h 50m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance > gain over the previous DirectRunner. We can do even better in multi-core > environments by supporting multi-process execution in the FnApiRunner, to > scale past Python GIL limitations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7437) Integration Test for BQ streaming inserts for streaming pipelines
[ https://issues.apache.org/jira/browse/BEAM-7437?focusedWorklogId=268714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268714 ] ASF GitHub Bot logged work on BEAM-7437: Author: ASF GitHub Bot Created on: 27/Jun/19 18:16 Start Date: 27/Jun/19 18:16 Worklog Time Spent: 10m Work Description: ttanay commented on issue #8934: [BEAM-7437] Add streaming flag to BQ streaming inserts IT test URL: https://github.com/apache/beam/pull/8934#issuecomment-506454961 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268714) Time Spent: 3h 50m (was: 3h 40m) > Integration Test for BQ streaming inserts for streaming pipelines > - > > Key: BEAM-7437 > URL: https://issues.apache.org/jira/browse/BEAM-7437 > Project: Beam > Issue Type: Test > Components: io-python-gcp >Affects Versions: 2.12.0 >Reporter: Tanay Tummalapalli >Assignee: Tanay Tummalapalli >Priority: Minor > Labels: test > Time Spent: 3h 50m > Remaining Estimate: 0h > > Integration Test for BigQuery Sink using Streaming Inserts for streaming > pipelines. > Integration tests currently exist for batch pipelines, it can also be added > for streaming pipelines using TestStream. This will be a precursor to the > failing integration test to be added for [BEAM-6611| > https://issues.apache.org/jira/browse/BEAM-6611]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7499) ReifyTest.test_window fails in DirectRunner due to 'assign_context.window should not be None.'
[
https://issues.apache.org/jira/browse/BEAM-7499?focusedWorklogId=268713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268713
]
ASF GitHub Bot logged work on BEAM-7499:
Author: ASF GitHub Bot
Created on: 27/Jun/19 18:15
Start Date: 27/Jun/19 18:15
Worklog Time Spent: 10m
Work Description: ttanay commented on issue #8866: [BEAM-7499] unskip
ReifyTest.test_window
URL: https://github.com/apache/beam/pull/8866#issuecomment-506454694
All tests passed :champagne:
@lukecwik PTAL
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268713)
Time Spent: 1h 50m (was: 1h 40m)
> ReifyTest.test_window fails in DirectRunner due to 'assign_context.window
> should not be None.'
> --
>
> Key: BEAM-7499
> URL: https://issues.apache.org/jira/browse/BEAM-7499
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core, test-failures
>Reporter: Luke Cwik
>Assignee: Pablo Estrada
>Priority: Minor
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
>
> [PR 8717|https://github.com/apache/beam/pull/8717] added
> ReifyWindow.test_window which fails on the DirectRunner.
> {code:java}
> ERROR:root:Exception at bundle
> ,
> due to an exception.
> Traceback (most recent call last):
> File "apache_beam/runners/direct/executor.py", line 343, in call
> finish_state)
> File "apache_beam/runners/direct/executor.py", line 380, in attempt_call
> evaluator.process_element(value)
> File "apache_beam/runners/direct/transform_evaluator.py", line 636, in
> process_element
> self.runner.process(element)
> File "apache_beam/runners/common.py", line 780, in
> apache_beam.runners.common.DoFnRunner.process
> def process(self, windowed_value):
> File "apache_beam/runners/common.py", line 784, in
> apache_beam.runners.common.DoFnRunner.process
> self._reraise_augmented(exn)
> File "apache_beam/runners/common.py", line 851, in
> apache_beam.runners.common.DoFnRunner._reraise_augmented
> raise_with_traceback(new_exn)
> File "apache_beam/runners/common.py", line 782, in
> apache_beam.runners.common.DoFnRunner.process
> return self.do_fn_invoker.invoke_process(windowed_value)
> File "apache_beam/runners/common.py", line 453, in
> apache_beam.runners.common.SimpleInvoker.invoke_process
> output_processor.process_outputs(
> File "apache_beam/runners/common.py", line 915, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> self.window_fn.assign(assign_context))
> File "apache_beam/transforms/util.py", line 557, in assign
> 'assign_context.window should not be None. '
> ValueError: assign_context.window should not be None. This might be due to a
> DoFn returning a TimestampedValue. [while running 'add_timestamps2']
> Traceback (most recent call last):
> File "apache_beam/transforms/util_test.py", line 501, in test_window
> assert_that(reified_pc, equal_to(expected), reify_windows=True)
> File "apache_beam/pipeline.py", line 426, in __exit__
> self.run().wait_until_finish()
> File "apache_beam/testing/test_pipeline.py", line 109, in run
> state = result.wait_until_finish()
> File "apache_beam/runners/direct/direct_runner.py", line 430, in
> wait_until_finish
> self._executor.await_completion()
> File "apache_beam/runners/direct/executor.py", line 400, in await_completion
> self._executor.await_completion()
> File "apache_beam/runners/direct/executor.py", line 446, in await_completion
> raise_(t, v, tb)
> File "apache_beam/runners/direct/executor.py", line 343, in call
> finish_state)
> File "apache_beam/runners/direct/executor.py", line 380, in attempt_call
> evaluator.process_element(value)
> File "apache_beam/runners/direct/transform_evaluator.py", line 636, in
> process_element
> self.runner.process(element)
> File "apache_beam/runners/common.py", line 780, in
> apache_beam.runners.common.DoFnRunner.process
> def process(self, windowed_value):
> File "apache_beam/runners/common.py", line 784, in
> apache_beam.runners.common.DoFnRunner.process
> self._reraise_augmented(exn)
> File "apache_beam/runners/common.py", line 851, in
> apache_beam.runners.common.DoFnRunner._reraise_augmented
> raise_with_traceback(new_exn)
> File "apache_beam/runners/common.py", line 782, in
> apache_beam.runners.common.DoFnRunner.process
> return self.do_fn_invoker.invoke_process(windowed_value)
> File
[jira] [Work logged] (BEAM-6611) A Python Sink for BigQuery with File Loads in Streaming
[ https://issues.apache.org/jira/browse/BEAM-6611?focusedWorklogId=268712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268712 ] ASF GitHub Bot logged work on BEAM-6611: Author: ASF GitHub Bot Created on: 27/Jun/19 18:14 Start Date: 27/Jun/19 18:14 Worklog Time Spent: 10m Work Description: ttanay commented on issue #8871: [BEAM-6611] BigQuery file loads in Streaming for Python SDK URL: https://github.com/apache/beam/pull/8871#issuecomment-506454294 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268712) Time Spent: 1h 10m (was: 1h) > A Python Sink for BigQuery with File Loads in Streaming > --- > > Key: BEAM-6611 > URL: https://issues.apache.org/jira/browse/BEAM-6611 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Tanay Tummalapalli >Priority: Major > Labels: gsoc, gsoc2019, mentor > Time Spent: 1h 10m > Remaining Estimate: 0h > > The Java SDK supports a bunch of methods for writing data into BigQuery, > while the Python SDK supports the following: > - Streaming inserts for streaming pipelines [As seen in [bigquery.py and > BigQueryWriteFn|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py#L649-L813]] > - File loads for batch pipelines [As implemented in [PR > 7655|https://github.com/apache/beam/pull/7655]] > Qucik and dirty early design doc: https://s.apache.org/beam-bqfl-py-streaming > The Java SDK also supports File Loads for Streaming pipelines [see BatchLoads > application|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1709-L1776]. > File loads have the advantage of being much cheaper than streaming inserts > (although they also are slower for the records to show up in the table). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7649) Match Python 3 warning messages in setup.py and __init.py__
[ https://issues.apache.org/jira/browse/BEAM-7649?focusedWorklogId=268705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268705 ] ASF GitHub Bot logged work on BEAM-7649: Author: ASF GitHub Bot Created on: 27/Jun/19 18:12 Start Date: 27/Jun/19 18:12 Worklog Time Spent: 10m Work Description: akedin commented on pull request #8956: [BEAM-7649] Match Python 3 warning message in __init__.py with the one in setup.py. URL: https://github.com/apache/beam/pull/8956 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268705) Time Spent: 20m (was: 10m) > Match Python 3 warning messages in setup.py and __init.py__ > --- > > Key: BEAM-7649 > URL: https://issues.apache.org/jira/browse/BEAM-7649 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Blocker > Fix For: 2.14.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268704
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 18:10
Start Date: 27/Jun/19 18:10
Worklog Time Spent: 10m
Work Description: chamikaramj commented on issue #8948: [BEAM-7548] fix
flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8948#issuecomment-506453101
Run Python_PVR_Flink PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268704)
Time Spent: 3.5h (was: 3h 20m)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response =
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268703=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268703
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 18:10
Start Date: 27/Jun/19 18:10
Worklog Time Spent: 10m
Work Description: chamikaramj commented on issue #8948: [BEAM-7548] fix
flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8948#issuecomment-506453053
Run Portable_Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268703)
Time Spent: 3h 20m (was: 3h 10m)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response =
[jira] [Work logged] (BEAM-6611) A Python Sink for BigQuery with File Loads in Streaming
[ https://issues.apache.org/jira/browse/BEAM-6611?focusedWorklogId=268702=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268702 ] ASF GitHub Bot logged work on BEAM-6611: Author: ASF GitHub Bot Created on: 27/Jun/19 18:09 Start Date: 27/Jun/19 18:09 Worklog Time Spent: 10m Work Description: ttanay commented on issue #8871: [BEAM-6611] BigQuery file loads in Streaming for Python SDK URL: https://github.com/apache/beam/pull/8871#issuecomment-506452684 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268702) Time Spent: 1h (was: 50m) > A Python Sink for BigQuery with File Loads in Streaming > --- > > Key: BEAM-6611 > URL: https://issues.apache.org/jira/browse/BEAM-6611 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Tanay Tummalapalli >Priority: Major > Labels: gsoc, gsoc2019, mentor > Time Spent: 1h > Remaining Estimate: 0h > > The Java SDK supports a bunch of methods for writing data into BigQuery, > while the Python SDK supports the following: > - Streaming inserts for streaming pipelines [As seen in [bigquery.py and > BigQueryWriteFn|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py#L649-L813]] > - File loads for batch pipelines [As implemented in [PR > 7655|https://github.com/apache/beam/pull/7655]] > Qucik and dirty early design doc: https://s.apache.org/beam-bqfl-py-streaming > The Java SDK also supports File Loads for Streaming pipelines [see BatchLoads > application|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1709-L1776]. > File loads have the advantage of being much cheaper than streaming inserts > (although they also are slower for the records to show up in the table). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268699=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268699
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 18:04
Start Date: 27/Jun/19 18:04
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #8948:
[BEAM-7548] fix flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8948#discussion_r298301936
##
File path: sdks/python/apache_beam/transforms/stats_test.py
##
@@ -154,13 +155,15 @@ def test_get_sample_size_from_est_error(self):
@unittest.skipIf(sys.version_info < (3, 0, 0),
Review comment:
This test failed again with py27, so I added skip back.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268699)
Time Spent: 3h 10m (was: 3h)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
[jira] [Work logged] (BEAM-7424) Retry HTTP 429 errors from GCS w/ exponential backoff when reading data
[ https://issues.apache.org/jira/browse/BEAM-7424?focusedWorklogId=268698=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268698 ] ASF GitHub Bot logged work on BEAM-7424: Author: ASF GitHub Bot Created on: 27/Jun/19 18:02 Start Date: 27/Jun/19 18:02 Worklog Time Spent: 10m Work Description: ihji commented on issue #8933: [BEAM-7424] Retry HTTP 429 errors from GCS URL: https://github.com/apache/beam/pull/8933#issuecomment-506450603 run python precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268698) Time Spent: 2.5h (was: 2h 20m) > Retry HTTP 429 errors from GCS w/ exponential backoff when reading data > --- > > Key: BEAM-7424 > URL: https://issues.apache.org/jira/browse/BEAM-7424 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, io-python-gcp, sdk-py-core >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Blocker > Fix For: 2.14.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > This has to be done for both Java and Python SDKs. > Seems like Java SDK already retries 429 errors w/o backoff (please verify): > [https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java#L185] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7547) StreamingDataflowWorker can observe inconsistent cache for stale work items
[ https://issues.apache.org/jira/browse/BEAM-7547?focusedWorklogId=268696=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268696 ] ASF GitHub Bot logged work on BEAM-7547: Author: ASF GitHub Bot Created on: 27/Jun/19 17:58 Start Date: 27/Jun/19 17:58 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #8842: [BEAM-7547] Avoid WindmillStateCache cache hits for stale work. URL: https://github.com/apache/beam/pull/8842#issuecomment-506449025 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268696) Time Spent: 2h (was: 1h 50m) > StreamingDataflowWorker can observe inconsistent cache for stale work items > --- > > Key: BEAM-7547 > URL: https://issues.apache.org/jira/browse/BEAM-7547 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Sam Whittle >Assignee: Sam Whittle >Priority: Minor > Time Spent: 2h > Remaining Estimate: 0h > > 1. Dataflow backend generates a work item with a cache token C. > 2. StreamingDataflowWorker receives the work item and reads the state using > C, it either hits the cache or performs a read. > 3. Dataflow backend sends a retry of the work item (possibly because it > thinks original work item never reached the StreamingDataflowWorker). > 4. StreamingDataflowWorker commits the work item and gets ack from dataflow > backend. It caches the state for the key using C. > 5. StreamingDataflowWorker receives the retried work item with cache token C. > It uses the cached state and causes possible user consistency failures > because the cache view is of after the work item completed processing. > Note that this will not cause corrupted Dataflow persistent state because the > commit of the retried work item using the inconsistent cache will fail. > However it may cause failures in user logic for example if they keep the set > of all seen items in state and throw an exception on duplicates which should > have been removed by an upstream stage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7586) Add Integration Test for MongoDbIO in python sdk
[ https://issues.apache.org/jira/browse/BEAM-7586?focusedWorklogId=268691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268691 ] ASF GitHub Bot logged work on BEAM-7586: Author: ASF GitHub Bot Created on: 27/Jun/19 17:54 Start Date: 27/Jun/19 17:54 Worklog Time Spent: 10m Work Description: y1chi commented on issue #8895: [BEAM-7586] Add Integration test for python mongodb io URL: https://github.com/apache/beam/pull/8895#issuecomment-506447539 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268691) Time Spent: 7h 50m (was: 7h 40m) > Add Integration Test for MongoDbIO in python sdk > - > > Key: BEAM-7586 > URL: https://issues.apache.org/jira/browse/BEAM-7586 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 7h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yifan zou reassigned BEAM-7648:
---
Assignee: yifan zou
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
> 18:11:49 at com.sun.proxy.$Proxy114.setRemoteUrl(Unknown Source)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:882)
>
[jira] [Work logged] (BEAM-7649) Match Python 3 warning messages in setup.py and __init.py__
[ https://issues.apache.org/jira/browse/BEAM-7649?focusedWorklogId=268687=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268687 ] ASF GitHub Bot logged work on BEAM-7649: Author: ASF GitHub Bot Created on: 27/Jun/19 17:50 Start Date: 27/Jun/19 17:50 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #8958: [BEAM-7649] Cherrypick PR-8956 onto 2.14.0 release branch URL: https://github.com/apache/beam/pull/8958#issuecomment-506446074 Test failures are noise, this change does not affect any tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268687) Time Spent: 10m Remaining Estimate: 0h > Match Python 3 warning messages in setup.py and __init.py__ > --- > > Key: BEAM-7649 > URL: https://issues.apache.org/jira/browse/BEAM-7649 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Blocker > Fix For: 2.14.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7649) Match Python 3 warning messages in setup.py and __init.py__
[ https://issues.apache.org/jira/browse/BEAM-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874377#comment-16874377 ] Valentyn Tymofieiev commented on BEAM-7649: --- cc: [~kedin] > Match Python 3 warning messages in setup.py and __init.py__ > --- > > Key: BEAM-7649 > URL: https://issues.apache.org/jira/browse/BEAM-7649 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Blocker > Fix For: 2.14.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7649) Match Python 3 warning messages in setup.py and __init.py__
Valentyn Tymofieiev created BEAM-7649: - Summary: Match Python 3 warning messages in setup.py and __init.py__ Key: BEAM-7649 URL: https://issues.apache.org/jira/browse/BEAM-7649 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Valentyn Tymofieiev Fix For: 2.14.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-1251) Python 3 Support
[ https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=268659=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268659 ] ASF GitHub Bot logged work on BEAM-1251: Author: ASF GitHub Bot Created on: 27/Jun/19 17:30 Start Date: 27/Jun/19 17:30 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #8956: [BEAM-1251] Match Python 3 warning message in __init__.py with the one in setup.py. URL: https://github.com/apache/beam/pull/8956#issuecomment-506439420 R: @akedin This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268659) Time Spent: 29h 10m (was: 29h) > Python 3 Support > > > Key: BEAM-1251 > URL: https://issues.apache.org/jira/browse/BEAM-1251 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Eyad Sibai >Assignee: Robbe >Priority: Major > Time Spent: 29h 10m > Remaining Estimate: 0h > > I have been trying to use google datalab with python3. As I see there are > several packages that does not support python3 yet which google datalab > depends on. This is one of them. > https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-1251) Python 3 Support
[ https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=268657=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268657 ] ASF GitHub Bot logged work on BEAM-1251: Author: ASF GitHub Bot Created on: 27/Jun/19 17:28 Start Date: 27/Jun/19 17:28 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #8956: [BEAM-1251] Match Python 3 warning message in __init__.py with the one in setup.py. URL: https://github.com/apache/beam/pull/8956 Beam has been offering Python 3 support for last 3 releases. Since then we made several improvements and established a solid test coverage. Aside from minor known issues, Beam is ready to be used on Python 3, therefore updating this warning message. Note that this change should have been included in https://github.com/apache/beam/pull/8867, that updates the warning message in setup.py, but was missed. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-7641) collect statistics about python ITs
[ https://issues.apache.org/jira/browse/BEAM-7641?focusedWorklogId=268651=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268651 ] ASF GitHub Bot logged work on BEAM-7641: Author: ASF GitHub Bot Created on: 27/Jun/19 17:22 Start Date: 27/Jun/19 17:22 Worklog Time Spent: 10m Work Description: udim commented on issue #8952: [BEAM-7641] Collect xunit statistics for Py ITs URL: https://github.com/apache/beam/pull/8952#issuecomment-506436645 R: @lukecwik @markflyhigh I know there are a lot of failures, but I assert that: 1. none of them are caused by this PR; 2. this PR will help us identify test failure frequency (for ITs). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 268651) Time Spent: 2h 40m (was: 2.5h) > collect statistics about python ITs > --- > > Key: BEAM-7641 > URL: https://issues.apache.org/jira/browse/BEAM-7641 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Currently ITs don't generate xunit (nosetests.xml) files. > Having this data will make it easier to see which tests failed in a > pre/postcommit run, and to tell if a particular test is flaky. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7648) apache-beam-jenkins-15 out of disk
Udi Meiri created BEAM-7648:
---
Summary: apache-beam-jenkins-15 out of disk
Key: BEAM-7648
URL: https://issues.apache.org/jira/browse/BEAM-7648
Project: Beam
Issue Type: Bug
Components: testing
Reporter: Udi Meiri
getting error like these:
{code:java}
18:11:49 ERROR: Error fetching remote repo 'origin'
18:11:49 hudson.plugins.git.GitException: Failed to fetch from
https://github.com/apache/beam.git
18:11:49at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
18:11:49at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
18:11:49at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
18:11:49at hudson.scm.SCM.checkout(SCM.java:504)
18:11:49at
hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
18:11:49at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
18:11:49at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
18:11:49at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
18:11:49at hudson.model.Run.execute(Run.java:1810)
18:11:49at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
18:11:49at
hudson.model.ResourceController.execute(ResourceController.java:97)
18:11:49at hudson.model.Executor.run(Executor.java:429)
18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
remote.origin.url https://github.com/apache/beam.git; returned status code 4:
18:11:49 stdout:
18:11:49 stderr: error: failed to write new configuration file
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
18:11:49
18:11:49at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
18:11:49at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
18:11:49at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
18:11:49at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
18:11:49at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
18:11:49at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
18:11:49at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
18:11:49at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
18:11:49at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
18:11:49at java.lang.reflect.Method.invoke(Method.java:498)
18:11:49at
hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
18:11:49at
hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
18:11:49at
hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
18:11:49at hudson.remoting.UserRequest.perform(UserRequest.java:212)
18:11:49at hudson.remoting.UserRequest.perform(UserRequest.java:54)
18:11:49at hudson.remoting.Request$2.run(Request.java:369)
18:11:49at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
18:11:49at java.util.concurrent.FutureTask.run(FutureTask.java:266)
18:11:49at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
18:11:49at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
18:11:49at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
18:11:49at java.lang.Thread.run(Thread.java:748)
18:11:49Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
call to JNLP4-connect connection from
103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
18:11:49at
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
18:11:49at
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
18:11:49at hudson.remoting.Channel.call(Channel.java:955)
18:11:49at
hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
18:11:49at com.sun.proxy.$Proxy114.setRemoteUrl(Unknown Source)
18:11:49at
org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
18:11:49at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:882)
18:11:49at
hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
18:11:49at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
18:11:49at hudson.scm.SCM.checkout(SCM.java:504)
18:11:49at
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268648=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268648
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 17:14
Start Date: 27/Jun/19 17:14
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #8948: [BEAM-7548] fix
flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8948#issuecomment-506433799
@tvalentyn I fixed all the points commented, can you please take a look when
you have time?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268648)
Time Spent: 3h (was: 2h 50m)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268644=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268644
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 17:10
Start Date: 27/Jun/19 17:10
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #8948: [BEAM-7548] fix
flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8948#issuecomment-506432730
Run Portable_Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268644)
Time Spent: 2h 50m (was: 2h 40m)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response =
[jira] [Work logged] (BEAM-7548) test_approximate_unique_global_by_error is flaky
[
https://issues.apache.org/jira/browse/BEAM-7548?focusedWorklogId=268643=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268643
]
ASF GitHub Bot logged work on BEAM-7548:
Author: ASF GitHub Bot
Created on: 27/Jun/19 17:10
Start Date: 27/Jun/19 17:10
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on issue #8948: [BEAM-7548] fix
flaky tests for ApproximateUnique
URL: https://github.com/apache/beam/pull/8948#issuecomment-506432665
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268643)
Time Spent: 2h 40m (was: 2.5h)
> test_approximate_unique_global_by_error is flaky
>
>
> Key: BEAM-7548
> URL: https://issues.apache.org/jira/browse/BEAM-7548
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Hannah Jiang
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The error happened on Jenkins in Python 3.5 suite, which currently uses
> Python 3.5.2 interpreter:
> {noformat}
> 11:57:47
> ==
> 11:57:47 ERROR: test_approximate_unique_global_by_error
> (apache_beam.transforms.stats_test.ApproximateUniqueTest)
> 11:57:47
> --
> 11:57:47 Traceback (most recent call last):
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/transforms/stats_test.py",
> line 236, in test_approximate_unique_global_by_error
> 11:57:47 pipeline.run()
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 11:57:47 else test_runner_api))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 11:57:47 self._options).run(False)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 11:57:47 return self.runner.run_pipeline(self, self._options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 128, in run_pipeline
> 11:57:47 return runner.run_pipeline(pipeline, options)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 289, in run_pipeline
> 11:57:47 default_environment=self._default_environment))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 293, in run_via_runner_api
> 11:57:47 return self.run_stages(*self.create_stages(pipeline_proto))
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 369, in run_stages
> 11:57:47 stage_context.safe_coders)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 531, in run_stage
> 11:57:47 data_input, data_output)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1235, in process_bundle
> 11:57:47 result_future =
> self._controller.control_handler.push(process_bundle)
> 11:57:47 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 851, in push
> 11:57:47 response =
[jira] [Commented] (BEAM-7589) Kinesis IO.write throws LimitExceededException
[
https://issues.apache.org/jira/browse/BEAM-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874353#comment-16874353
]
Brachi Packter commented on BEAM-7589:
--
Great! Waiting it to be merged. And then I'll test it.
On Thu, Jun 27, 2019, 7:59 PM Alexey Romanenko (JIRA)
> Kinesis IO.write throws LimitExceededException
> --
>
> Key: BEAM-7589
> URL: https://issues.apache.org/jira/browse/BEAM-7589
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
>Affects Versions: 2.11.0
>Reporter: Anton Kedin
>Assignee: Alexey Romanenko
>Priority: Major
> Fix For: 2.15.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Follow up from https://issues.apache.org/jira/browse/BEAM-7357:
>
>
> Brachi Packter added a comment - 13/Jun/19 09:05
> [~aromanenko] I think I find what makes the shard map update now.
> You create a producer per bundle (in SetUp function) and if I multiply it by
> the number of workers, this gives huge amount of producers, I belive this
> make the "update shard map" call.
> If I copy your code and create *one* producer ** for every wroker, then this
> error disappear.
> Can you just remove the producer creation from setUp method, and move it to
> some static field in the class, that created once the class is initiated.
> See similar issue that was with JDBCIO, connection pool was created per setup
> method, and we moved it to be a static member, and then we will have one pool
> for JVM. ask [~iemejia] for more detail.
>
> Alexey Romanenko added a comment -14/Jun/19 14:31- edited
>
> [~brachi_packter] What kind of error do you have in this case? Could you
> post an error stacktrace / exception message?
> Also, it would be helpful (if it's possible) if you could provide more
> details about your environment and pipeline, like what is your pipeline
> topology, which runner do you use, number of workers in your cluster, etc.
> For now, I can't reproduce it on my side, so all additional info will be
> helpful.
>
> Brachi Packter added a comment - 16/Jun/19 06:44
> I get same Same error:
> {code:java}
> [0x1728][0x7f13ed4c4700] [error] [shard_map.cc:150] Shard map update
> for stream "**" failed. Code: LimitExceededException Message: Rate exceeded
> for stream poc-test under account **.; retrying in 5062 ms
> {code}
> I'm not seeing full stack trace, but can see in log also this:
> {code:java}
> [2019-06-13 08:29:09.427018] [0x07e1][0x7f8d508d3700] [warning] [AWS
> Log: WARN](AWSErrorMarshaller)Encountered AWSError Throttling Rate exceeded
> {code}
> More details:
> I'm using DataFlow runner, java SDK 2.11.
> 60 workers initally, (with auto scalling and also with flag
> "enableStreamingEngine")
> Normally, I'm producing 4-5k per second, but when I have latency, this can be
> even multiply by 3-4 times.
> When I'm starting the DataFlow job I have latency, so I produce more data,
> and I fail immediately.
> Also, I have consumers, 3rd party tool, I know that they call describe stream
> each 30 seconds.
> My job pipeline, running on GCP, reading data from PubSub, it read around
> 20,000 record per second (in regular time, and in latency time even 100,000
> records per second) , it does many aggregation and counting base on some
> diamnesions (Using Beam sql) , This is done for 1 minutes window slide, and
> wrting the result of aggregations to Kinesis stream.
> My stream has 10 shards, and my partition key logic is generating UUid per
> each record:
> UUID.randomUUID().toString()
> Hope this gave you some more context on my problem.
> Another suggestion I have, can you try fix the issue as I suggest and provide
> me some specific version for testing? without merging it to master? (I would
> di it myself, but I had truobles building locally the hue repository of
> apache beam..)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
