[jira] [Commented] (BEAM-7049) Merge multiple input to one BeamUnionRel
[ https://issues.apache.org/jira/browse/BEAM-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929007#comment-16929007 ] Rui Wang commented on BEAM-7049: I can then observe that there are 5 inputs at here: https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase.java#L61 And what left is how to make N input go through CoGBK? > Merge multiple input to one BeamUnionRel > > > Key: BEAM-7049 > URL: https://issues.apache.org/jira/browse/BEAM-7049 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: sridhar Reddy >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > BeamUnionRel assumes inputs are two and rejects more. So `a UNION b UNION c` > will have to be created as UNION(a, UNION(b, c)) and have two shuffles. If > BeamUnionRel can handle multiple shuffles, we will have only one shuffle -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311884&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311884 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 13/Sep/19 06:06 Start Date: 13/Sep/19 06:06 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-531110906 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311884) Time Spent: 10h 10m (was: 10h) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 10h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311883&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311883 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 13/Sep/19 06:06 Start Date: 13/Sep/19 06:06 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-531110888 Run Spark Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311883) Time Spent: 10h (was: 9h 50m) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 10h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311885&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311885 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 13/Sep/19 06:06 Start Date: 13/Sep/19 06:06 Worklog Time Spent: 10m Work Description: vectorijk commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-531110946 Run Direct Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311885) Time Spent: 10h 20m (was: 10h 10m) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 10h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311880&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311880 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 13/Sep/19 05:55 Start Date: 13/Sep/19 05:55 Worklog Time Spent: 10m Work Description: vectorijk commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-531108625 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311880) Time Spent: 9h 50m (was: 9h 40m) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 9h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311879&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311879 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 13/Sep/19 05:54 Start Date: 13/Sep/19 05:54 Worklog Time Spent: 10m Work Description: vectorijk commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-531108567 Run Spark Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311879) Time Spent: 9h 40m (was: 9.5h) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 9h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-7049) Merge multiple input to one BeamUnionRel
[
https://issues.apache.org/jira/browse/BEAM-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928996#comment-16928996
]
Rui Wang commented on BEAM-7049:
I used query
"SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL
SELECT 5" to test and also enabled the rule.
I got the plan:
{code:java}
INFO: BEAMPlan>
BeamUnionRel(all=[true])
BeamCalcRel(expr#0=[{inputs}], expr#1=[1], EXPR$0=[$t1])
BeamValuesRel(tuples=[[{ 0 }]])
BeamCalcRel(expr#0=[{inputs}], expr#1=[2], EXPR$0=[$t1])
BeamValuesRel(tuples=[[{ 0 }]])
BeamCalcRel(expr#0=[{inputs}], expr#1=[3], EXPR$0=[$t1])
BeamValuesRel(tuples=[[{ 0 }]])
BeamCalcRel(expr#0=[{inputs}], expr#1=[4], EXPR$0=[$t1])
BeamValuesRel(tuples=[[{ 0 }]])
BeamCalcRel(expr#0=[{inputs}], expr#1=[5], EXPR$0=[$t1])
BeamValuesRel(tuples=[[{ 0 }]])
{code}
which is expected as BeamUnionRel has a list of inputs with 5 BeamCalcRel(and
each BeamCalcRel will be convert to a PCollection of Row)
> Merge multiple input to one BeamUnionRel
>
>
> Key: BEAM-7049
> URL: https://issues.apache.org/jira/browse/BEAM-7049
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: sridhar Reddy
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> BeamUnionRel assumes inputs are two and rejects more. So `a UNION b UNION c`
> will have to be created as UNION(a, UNION(b, c)) and have two shuffles. If
> BeamUnionRel can handle multiple shuffles, we will have only one shuffle
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (BEAM-8227) Move private IP dependency validation test into a Beam project test
Hannah Jiang created BEAM-8227: -- Summary: Move private IP dependency validation test into a Beam project test Key: BEAM-8227 URL: https://issues.apache.org/jira/browse/BEAM-8227 Project: Beam Issue Type: Improvement Components: testing Reporter: Hannah Jiang Assignee: Hannah Jiang Move private IP dependency validation test into a Beam project test (rather than internal Dataflow) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311856&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311856 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 13/Sep/19 03:04 Start Date: 13/Sep/19 03:04 Worklog Time Spent: 10m Work Description: vectorijk commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-531081137 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311856) Time Spent: 9.5h (was: 9h 20m) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 9.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-3342) Create a Cloud Bigtable IO connector for Python
[
https://issues.apache.org/jira/browse/BEAM-3342?focusedWorklogId=311853&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311853
]
ASF GitHub Bot logged work on BEAM-3342:
Author: ASF GitHub Bot
Created on: 13/Sep/19 02:56
Start Date: 13/Sep/19 02:56
Worklog Time Spent: 10m
Work Description: mf2199 commented on pull request #8457: [BEAM-3342]
Create a Cloud Bigtable IO connector for Python
URL: https://github.com/apache/beam/pull/8457#discussion_r324020715
##
File path: sdks/python/apache_beam/io/gcp/bigtableio_it_test.py
##
@@ -0,0 +1,189 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+""" Integration test for GCP Bigtable testing."""
+from __future__ import absolute_import
+
+import argparse
+import datetime
+import logging
+import random
+import string
+import time
+import unittest
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam.metrics.metric import MetricsFilter
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing.util import assert_that, equal_to
+from apache_beam.transforms.combiners import Count
+
+try:
+ from google.cloud.bigtable import enums, row, column_family, Client
+except ImportError:
+ Client = None
+
+import bigtableio
+
+class GenerateTestRows(beam.PTransform):
+ """ A PTransform to generate dummy rows to write to a Bigtable Table.
+
+ A PTransform that generates a list of `DirectRow` and writes it to a
Bigtable Table.
+ """
+ def __init__(self):
+super(self.__class__, self).__init__()
+self.beam_options = {'project_id': PROJECT_ID,
+ 'instance_id': INSTANCE_ID,
+ 'table_id': TABLE_ID}
+
+ def _generate(self):
+for i in range(ROW_COUNT):
+ key = "key_%s" % ('{0:012}'.format(i))
+ test_row = row.DirectRow(row_key=key)
+ value = ''.join(random.choice(LETTERS_AND_DIGITS) for _ in
range(CELL_SIZE))
+ for j in range(COLUMN_COUNT):
+test_row.set_cell(column_family_id=COLUMN_FAMILY_ID,
+ column=('field%s' % j).encode('utf-8'),
+ value=value,
+ timestamp=datetime.datetime.now())
+ yield test_row
+
+ def expand(self, pvalue):
+return (pvalue
+| beam.Create(self._generate())
+|
bigtableio.WriteToBigTable(project_id=self.beam_options['project_id'],
+
instance_id=self.beam_options['instance_id'],
+
table_id=self.beam_options['table_id']))
+
+@unittest.skipIf(Client is None, 'GCP Bigtable dependencies are not installed')
+class BigtableIOTest(unittest.TestCase):
+ """ Bigtable IO Connector Test
+
+ This tests the connector both ways, first writing rows to a new table, then
reading them and comparing the counters
+ """
+ def setUp(self):
+self.result = None
+self.table = Client(project=PROJECT_ID, admin=True)\
+.instance(instance_id=INSTANCE_ID)\
+.table(TABLE_ID)
+
+if not self.table.exists():
+ column_families = {COLUMN_FAMILY_ID: column_family.MaxVersionsGCRule(2)}
+ self.table.create(column_families=column_families)
+ logging.info('Table {} has been created!'.format(TABLE_ID))
+
+ @attr('IT')
+ def test_bigtable_io(self):
+print 'Project ID: ', PROJECT_ID
+print 'Instance ID:', INSTANCE_ID
+print 'Table ID: ', TABLE_ID
Review comment:
Changed to `logging.info(...)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311853)
Time Spent: 38h 20m (was: 38h 10m)
> Create a Cloud Bigtable IO connector for Python
> ---
>
[jira] [Work logged] (BEAM-3342) Create a Cloud Bigtable IO connector for Python
[ https://issues.apache.org/jira/browse/BEAM-3342?focusedWorklogId=311851&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311851 ] ASF GitHub Bot logged work on BEAM-3342: Author: ASF GitHub Bot Created on: 13/Sep/19 02:38 Start Date: 13/Sep/19 02:38 Worklog Time Spent: 10m Work Description: mf2199 commented on pull request #8457: [BEAM-3342] Create a Cloud Bigtable IO connector for Python URL: https://github.com/apache/beam/pull/8457#discussion_r324018337 ## File path: sdks/python/apache_beam/io/gcp/bigtableio_it_test.py ## @@ -0,0 +1,189 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +""" Integration test for GCP Bigtable testing.""" +from __future__ import absolute_import + +import argparse +import datetime +import logging +import random +import string +import time +import unittest +from nose.plugins.attrib import attr + +import apache_beam as beam +from apache_beam.metrics.metric import MetricsFilter +from apache_beam.options.pipeline_options import PipelineOptions +from apache_beam.runners.runner import PipelineState +from apache_beam.testing.util import assert_that, equal_to +from apache_beam.transforms.combiners import Count + +try: + from google.cloud.bigtable import enums, row, column_family, Client +except ImportError: + Client = None + +import bigtableio Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311851) Time Spent: 38h 10m (was: 38h) > Create a Cloud Bigtable IO connector for Python > --- > > Key: BEAM-3342 > URL: https://issues.apache.org/jira/browse/BEAM-3342 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Solomon Duskis >Assignee: Solomon Duskis >Priority: Major > Time Spent: 38h 10m > Remaining Estimate: 0h > > I would like to create a Cloud Bigtable python connector. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (BEAM-8226) Python Streaming Pipeline getting stuck in dataflow
[ https://issues.apache.org/jira/browse/BEAM-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Goenka updated BEAM-8226: --- Fix Version/s: 2.16.0 > Python Streaming Pipeline getting stuck in dataflow > --- > > Key: BEAM-8226 > URL: https://issues.apache.org/jira/browse/BEAM-8226 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness, sdk-py-harness >Reporter: Ankur Goenka >Priority: Major > Fix For: 2.16.0 > > > Python streaming pipeline are getting stuck with following error when runing > on dataflow > > Relevant thread stack > --- Threads (4): [Thread[Thread-19,1,main], Thread[Thread-20,1,main], > Thread[Thread-21,1,main], Thread[Thread-22,1,main]] State: WAITING stack: --- > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > > org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.maybeWait(RemoteGrpcPortWriteOperation.java:175) > > org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.process(RemoteGrpcPortWriteOperation.java:196) > > org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49) > > org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201) > > org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159) > > org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77) > > org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1316) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:149) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1049) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > java.lang.Thread.run(Thread.java:745) > For Python > --- Thread #139819623634688 name: ThreadPoolExecutor-1_0 --- > File "/usr/local/lib/python3.6/threading.py", line 884, in _bootstrap > self._bootstrap_inner() > File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner > self.run() > File "/usr/local/lib/python3.6/threading.py", line 864, in run > self._target(*self._args, **self._kwargs) > File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 69, in > _worker > work_item.run() > File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in > run > result = self.fn(*self.args, **self.kwargs) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 191, in task > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 158, in _execute > response = task() > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 191, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 343, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 369, in process_bundle > bundle_processor.process_bundle(instruction_id)) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 661, in process_bundle > instruction_id, expected_transforms): > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/data_plane.py", > line 213, in input_elements > data = received.get(timeout=1) > File "/usr/local/lib/python3.6/queue.py", line 173, in get > self.not_empty.wait(remaining) > File "/usr/local/lib/python3.6/threading.py", line 299, in wait > gotit = waiter.acquire(True, timeout) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8226) Python Streaming Pipeline getting stuck in dataflow
[ https://issues.apache.org/jira/browse/BEAM-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928971#comment-16928971 ] Ankur Goenka commented on BEAM-8226: cc: [~markflyhigh] [~yichi] [~altay] [~tvalentyn] > Python Streaming Pipeline getting stuck in dataflow > --- > > Key: BEAM-8226 > URL: https://issues.apache.org/jira/browse/BEAM-8226 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness, sdk-py-harness >Reporter: Ankur Goenka >Priority: Blocker > Fix For: 2.16.0 > > > Python streaming pipeline are getting stuck with following error when runing > on dataflow > > Relevant thread stack > --- Threads (4): [Thread[Thread-19,1,main], Thread[Thread-20,1,main], > Thread[Thread-21,1,main], Thread[Thread-22,1,main]] State: WAITING stack: --- > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > > org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.maybeWait(RemoteGrpcPortWriteOperation.java:175) > > org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.process(RemoteGrpcPortWriteOperation.java:196) > > org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49) > > org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201) > > org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159) > > org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77) > > org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1316) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:149) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1049) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > java.lang.Thread.run(Thread.java:745) > For Python > --- Thread #139819623634688 name: ThreadPoolExecutor-1_0 --- > File "/usr/local/lib/python3.6/threading.py", line 884, in _bootstrap > self._bootstrap_inner() > File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner > self.run() > File "/usr/local/lib/python3.6/threading.py", line 864, in run > self._target(*self._args, **self._kwargs) > File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 69, in > _worker > work_item.run() > File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in > run > result = self.fn(*self.args, **self.kwargs) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 191, in task > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 158, in _execute > response = task() > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 191, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 343, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 369, in process_bundle > bundle_processor.process_bundle(instruction_id)) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 661, in process_bundle > instruction_id, expected_transforms): > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/data_plane.py", > line 213, in input_elements > data = received.get(timeout=1) > File "/usr/local/lib/python3.6/queue.py", line 173, in get > self.not_empty.wait(remaining) > File "/usr/local/lib/python3.6/threading.py", line 299, in wait > gotit = waiter.acquire(True, timeout) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (BEAM-8226) Python Streaming Pipeline getting stuck in dataflow
[ https://issues.apache.org/jira/browse/BEAM-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Goenka updated BEAM-8226: --- Priority: Blocker (was: Major) > Python Streaming Pipeline getting stuck in dataflow > --- > > Key: BEAM-8226 > URL: https://issues.apache.org/jira/browse/BEAM-8226 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness, sdk-py-harness >Reporter: Ankur Goenka >Priority: Blocker > Fix For: 2.16.0 > > > Python streaming pipeline are getting stuck with following error when runing > on dataflow > > Relevant thread stack > --- Threads (4): [Thread[Thread-19,1,main], Thread[Thread-20,1,main], > Thread[Thread-21,1,main], Thread[Thread-22,1,main]] State: WAITING stack: --- > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > > org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.maybeWait(RemoteGrpcPortWriteOperation.java:175) > > org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.process(RemoteGrpcPortWriteOperation.java:196) > > org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49) > > org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201) > > org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159) > > org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77) > > org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1316) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:149) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1049) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > java.lang.Thread.run(Thread.java:745) > For Python > --- Thread #139819623634688 name: ThreadPoolExecutor-1_0 --- > File "/usr/local/lib/python3.6/threading.py", line 884, in _bootstrap > self._bootstrap_inner() > File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner > self.run() > File "/usr/local/lib/python3.6/threading.py", line 864, in run > self._target(*self._args, **self._kwargs) > File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 69, in > _worker > work_item.run() > File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in > run > result = self.fn(*self.args, **self.kwargs) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 191, in task > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 158, in _execute > response = task() > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 191, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 343, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 369, in process_bundle > bundle_processor.process_bundle(instruction_id)) > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 661, in process_bundle > instruction_id, expected_transforms): > File > "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/data_plane.py", > line 213, in input_elements > data = received.get(timeout=1) > File "/usr/local/lib/python3.6/queue.py", line 173, in get > self.not_empty.wait(remaining) > File "/usr/local/lib/python3.6/threading.py", line 299, in wait > gotit = waiter.acquire(True, timeout) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (BEAM-8226) Python Streaming Pipeline getting stuck in dataflow
Ankur Goenka created BEAM-8226: -- Summary: Python Streaming Pipeline getting stuck in dataflow Key: BEAM-8226 URL: https://issues.apache.org/jira/browse/BEAM-8226 Project: Beam Issue Type: Bug Components: sdk-java-harness, sdk-py-harness Reporter: Ankur Goenka Python streaming pipeline are getting stuck with following error when runing on dataflow Relevant thread stack --- Threads (4): [Thread[Thread-19,1,main], Thread[Thread-20,1,main], Thread[Thread-21,1,main], Thread[Thread-22,1,main]] State: WAITING stack: --- sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.maybeWait(RemoteGrpcPortWriteOperation.java:175) org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.process(RemoteGrpcPortWriteOperation.java:196) org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159) org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77) org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1316) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:149) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1049) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) For Python --- Thread #139819623634688 name: ThreadPoolExecutor-1_0 --- File "/usr/local/lib/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/local/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 69, in _worker work_item.run() File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", line 191, in task self._execute(lambda: worker.do_instruction(work), work) File "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", line 158, in _execute response = task() File "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", line 191, in self._execute(lambda: worker.do_instruction(work), work) File "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", line 343, in do_instruction request.instruction_id) File "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py", line 369, in process_bundle bundle_processor.process_bundle(instruction_id)) File "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py", line 661, in process_bundle instruction_id, expected_transforms): File "/usr/local/lib/python3.6/site-packages/apache_beam/runners/worker/data_plane.py", line 213, in input_elements data = received.get(timeout=1) File "/usr/local/lib/python3.6/queue.py", line 173, in get self.not_empty.wait(remaining) File "/usr/local/lib/python3.6/threading.py", line 299, in wait gotit = waiter.acquire(True, timeout) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8224) The way we inspect arguments in Python does not work for lambda *_: ...
[ https://issues.apache.org/jira/browse/BEAM-8224?focusedWorklogId=311842&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311842 ] ASF GitHub Bot logged work on BEAM-8224: Author: ASF GitHub Bot Created on: 13/Sep/19 01:56 Start Date: 13/Sep/19 01:56 Worklog Time Spent: 10m Work Description: udim commented on issue #9563: [BEAM-8224] Fix bug in _fn_takes_side_inputs URL: https://github.com/apache/beam/pull/9563#issuecomment-531070126 run python 2 postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311842) Time Spent: 40m (was: 0.5h) > The way we inspect arguments in Python does not work for lambda *_: ... > --- > > Key: BEAM-8224 > URL: https://issues.apache.org/jira/browse/BEAM-8224 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Yueyang Qiu >Assignee: Udi Meiri >Priority: Blocker > Fix For: 2.16.0 > > Time Spent: 40m > Remaining Estimate: 0h > > We need to make it work and create a test for this case. > > Assigning to Udi who authored > [https://github.com/apache/beam/pull/9283|https://www.google.com/url?q=https://github.com/apache/beam/pull/9283&sa=D&usg=AFQjCNETtLzhe9jPfGzLHJb2N4Sle-FuNw]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8224) The way we inspect arguments in Python does not work for lambda *_: ...
[ https://issues.apache.org/jira/browse/BEAM-8224?focusedWorklogId=311843&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311843 ] ASF GitHub Bot logged work on BEAM-8224: Author: ASF GitHub Bot Created on: 13/Sep/19 01:56 Start Date: 13/Sep/19 01:56 Worklog Time Spent: 10m Work Description: udim commented on issue #9563: [BEAM-8224] Fix bug in _fn_takes_side_inputs URL: https://github.com/apache/beam/pull/9563#issuecomment-531070140 run python 3.7 postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311843) Time Spent: 50m (was: 40m) > The way we inspect arguments in Python does not work for lambda *_: ... > --- > > Key: BEAM-8224 > URL: https://issues.apache.org/jira/browse/BEAM-8224 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Yueyang Qiu >Assignee: Udi Meiri >Priority: Blocker > Fix For: 2.16.0 > > Time Spent: 50m > Remaining Estimate: 0h > > We need to make it work and create a test for this case. > > Assigning to Udi who authored > [https://github.com/apache/beam/pull/9283|https://www.google.com/url?q=https://github.com/apache/beam/pull/9283&sa=D&usg=AFQjCNETtLzhe9jPfGzLHJb2N4Sle-FuNw]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8224) The way we inspect arguments in Python does not work for lambda *_: ...
[ https://issues.apache.org/jira/browse/BEAM-8224?focusedWorklogId=311841&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311841 ] ASF GitHub Bot logged work on BEAM-8224: Author: ASF GitHub Bot Created on: 13/Sep/19 01:55 Start Date: 13/Sep/19 01:55 Worklog Time Spent: 10m Work Description: udim commented on issue #9563: [BEAM-8224] Fix bug in _fn_takes_side_inputs URL: https://github.com/apache/beam/pull/9563#issuecomment-531070097 run pytthon 2 postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311841) Time Spent: 0.5h (was: 20m) > The way we inspect arguments in Python does not work for lambda *_: ... > --- > > Key: BEAM-8224 > URL: https://issues.apache.org/jira/browse/BEAM-8224 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Yueyang Qiu >Assignee: Udi Meiri >Priority: Blocker > Fix For: 2.16.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > We need to make it work and create a test for this case. > > Assigning to Udi who authored > [https://github.com/apache/beam/pull/9283|https://www.google.com/url?q=https://github.com/apache/beam/pull/9283&sa=D&usg=AFQjCNETtLzhe9jPfGzLHJb2N4Sle-FuNw]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8224) The way we inspect arguments in Python does not work for lambda *_: ...
[ https://issues.apache.org/jira/browse/BEAM-8224?focusedWorklogId=311840&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311840 ] ASF GitHub Bot logged work on BEAM-8224: Author: ASF GitHub Bot Created on: 13/Sep/19 01:52 Start Date: 13/Sep/19 01:52 Worklog Time Spent: 10m Work Description: udim commented on issue #9563: [BEAM-8224] Fix bug in _fn_takes_side_inputs URL: https://github.com/apache/beam/pull/9563#issuecomment-531069557 R: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311840) Time Spent: 20m (was: 10m) > The way we inspect arguments in Python does not work for lambda *_: ... > --- > > Key: BEAM-8224 > URL: https://issues.apache.org/jira/browse/BEAM-8224 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Yueyang Qiu >Assignee: Udi Meiri >Priority: Blocker > Fix For: 2.16.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We need to make it work and create a test for this case. > > Assigning to Udi who authored > [https://github.com/apache/beam/pull/9283|https://www.google.com/url?q=https://github.com/apache/beam/pull/9283&sa=D&usg=AFQjCNETtLzhe9jPfGzLHJb2N4Sle-FuNw]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8224) The way we inspect arguments in Python does not work for lambda *_: ...
[ https://issues.apache.org/jira/browse/BEAM-8224?focusedWorklogId=311838&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311838 ] ASF GitHub Bot logged work on BEAM-8224: Author: ASF GitHub Bot Created on: 13/Sep/19 01:50 Start Date: 13/Sep/19 01:50 Worklog Time Spent: 10m Work Description: udim commented on pull request #9563: [BEAM-8224] Fix bug in _fn_takes_side_inputs URL: https://github.com/apache/beam/pull/9563 Bug was introduced in the conversion to inspect.signature, and manifests as assigning a wrapper function that accepts a single positional argument when the wrapped function can accept an arbitrary number. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[!
[jira] [Updated] (BEAM-8224) The way we inspect arguments in Python does not work for lambda *_: ...
[ https://issues.apache.org/jira/browse/BEAM-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri updated BEAM-8224: Status: Open (was: Triage Needed) > The way we inspect arguments in Python does not work for lambda *_: ... > --- > > Key: BEAM-8224 > URL: https://issues.apache.org/jira/browse/BEAM-8224 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Yueyang Qiu >Assignee: Udi Meiri >Priority: Blocker > Fix For: 2.16.0 > > > We need to make it work and create a test for this case. > > Assigning to Udi who authored > [https://github.com/apache/beam/pull/9283|https://www.google.com/url?q=https://github.com/apache/beam/pull/9283&sa=D&usg=AFQjCNETtLzhe9jPfGzLHJb2N4Sle-FuNw]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311836&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311836 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 13/Sep/19 00:40 Start Date: 13/Sep/19 00:40 Worklog Time Spent: 10m Work Description: vectorijk commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-531057820 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311836) Time Spent: 9h 20m (was: 9h 10m) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 9h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (BEAM-7911) test_corrupted_file test flaky
[
https://issues.apache.org/jira/browse/BEAM-7911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri resolved BEAM-7911.
-
Fix Version/s: Not applicable
Resolution: Fixed
> test_corrupted_file test flaky
> --
>
> Key: BEAM-7911
> URL: https://issues.apache.org/jira/browse/BEAM-7911
> Project: Beam
> Issue Type: Bug
> Components: io-py-avro, test-failures
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Looks like a flake:
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7924/consoleFull
> 11:22:24
> 11:22:24
> ==
> 11:22:24 ERROR: test_corrupted_file (apache_beam.io.avroio_test.TestFastAvro)
> 11:22:24
> --
> 11:22:24 Traceback (most recent call last):
> 11:22:24 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/avroio_test.py",
> line 380, in test_corrupted_file
> 11:22:24 self.assertEqual(0, exn.exception.message.find('Unexpected sync
> marker'))
> 11:22:24 AttributeError: '_AssertRaisesContext' object has no attribute
> 'exception'
> 11:22:24 >> begin captured logging <<
>
> 11:22:24 apache_beam.io.filesystem: DEBUG: translate_pattern:
> '/tmp/tmpckgw8mk6' -> '\\/tmp\\/tmpckgw8mk6'
> 11:22:24 - >> end captured logging <<
> -
> 11:22:24
> 11:22:24
> --
> 11:22:24 XML:
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/nosetests.xml
> 11:22:24
> --
> 11:22:24 Ran 2455 tests in 1297.267s
> 11:22:24
> 11:22:24 FAILED (SKIP=546, errors=1)
> 11:22:24 ERROR: InvocationError for command
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36/py36/bin/python
> setup.py nosetests (exited with code 1)
> 11:22:24 py36 run-test-post: commands[0] |
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
> 11:22:24 ___ summary
>
> 11:22:24 ERROR: py36: commands failed
> 11:22:24
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-3713) Consider moving away from nose to nose2 or pytest.
[ https://issues.apache.org/jira/browse/BEAM-3713?focusedWorklogId=311834&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311834 ] ASF GitHub Bot logged work on BEAM-3713: Author: ASF GitHub Bot Created on: 13/Sep/19 00:24 Start Date: 13/Sep/19 00:24 Worklog Time Spent: 10m Work Description: udim commented on pull request #9514: [BEAM-3713] Convert ITs to not use save_main_session URL: https://github.com/apache/beam/pull/9514#discussion_r323999201 ## File path: sdks/python/apache_beam/examples/complete/distribopt.py ## @@ -314,7 +314,7 @@ def format_output(element): return result -def run(argv=None): +def run(argv=None, save_main_session=True): Review comment: I was thinking today (but haven't had time to check) that since save_main_session is also a command line option, it could be passed in `argv`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311834) Time Spent: 6.5h (was: 6h 20m) > Consider moving away from nose to nose2 or pytest. > -- > > Key: BEAM-3713 > URL: https://issues.apache.org/jira/browse/BEAM-3713 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: Robert Bradshaw >Assignee: Udi Meiri >Priority: Minor > Time Spent: 6.5h > Remaining Estimate: 0h > > Per > [https://nose.readthedocs.io/en/latest/|https://nose.readthedocs.io/en/latest/,] > , nose is in maintenance mode. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8221) NullPointerException in reading from non-existent Kafka topic
[
https://issues.apache.org/jira/browse/BEAM-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928958#comment-16928958
]
Chandan Kumar commented on BEAM-8221:
-
I can work on this :)
> NullPointerException in reading from non-existent Kafka topic
> -
>
> Key: BEAM-8221
> URL: https://issues.apache.org/jira/browse/BEAM-8221
> Project: Beam
> Issue Type: Bug
> Components: io-java-kafka
>Reporter: Chandan Kumar
>Priority: Major
>
> Get this exception when launch apache beam job, which tries to read from
> non-existent Kafka topic:
> {code:java}
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.NullPointerException
> at
> org.apache.beam.runners.dataflow.ReadTranslator.translateReadHelper(ReadTranslator.java:54)
> at
> org.apache.beam.runners.dataflow.DataflowRunner$StreamingUnboundedRead$ReadWithIdsTranslator.translate(DataflowRunner.java:1508)
> at
> org.apache.beam.runners.dataflow.DataflowRunner$StreamingUnboundedRead$ReadWithIdsTranslator.translate(DataflowRunner.java:1505)
> at
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:475)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
> at
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
> at
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
> at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
> at
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:414)
> at
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:173)
> at
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:737)
> at
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:179)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> at
> com.paloaltonetworks.cortex.replication.ingest.ReplicationIngestor.run(ReplicationIngestor.java:109)
> at
> com.paloaltonetworks.cortex.replication.ingest.ReplicationIngestor.main(ReplicationIngestor.java:64)
> Caused by: java.lang.NullPointerException
> at
> org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.split(KafkaUnboundedSource.java:66)
> at
> org.apache.beam.runners.dataflow.internal.CustomSources.serializeToCloudSource(CustomSources.java:87)
> at
> org.apache.beam.runners.dataflow.ReadTranslator.translateReadHelper(ReadTranslator.java:51)
> ... 18 more {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (BEAM-7049) Merge multiple input to one BeamUnionRel
[ https://issues.apache.org/jira/browse/BEAM-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928956#comment-16928956 ] sridhar Reddy commented on BEAM-7049: - Sounds good! > Merge multiple input to one BeamUnionRel > > > Key: BEAM-7049 > URL: https://issues.apache.org/jira/browse/BEAM-7049 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: sridhar Reddy >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > BeamUnionRel assumes inputs are two and rejects more. So `a UNION b UNION c` > will have to be created as UNION(a, UNION(b, c)) and have two shuffles. If > BeamUnionRel can handle multiple shuffles, we will have only one shuffle -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (BEAM-8156) Finish migration to standard Python typing
[ https://issues.apache.org/jira/browse/BEAM-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri updated BEAM-8156: Status: Open (was: Triage Needed) > Finish migration to standard Python typing > -- > > Key: BEAM-8156 > URL: https://issues.apache.org/jira/browse/BEAM-8156 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Udi Meiri >Priority: Major > Original Estimate: 504h > Time Spent: 1h 40m > Remaining Estimate: 502h 20m > > We should migrate all Python uses of types to the standard typing module, and > make the typehints.* ones aliases of the Python ones. > > There are three places where we use custom typehints behavior: > (1) is_compatible_with > (2) bind_type_variables/match_type_variables > (3) trivial type inference. > > I would propose that each of these be adapted to a (internal) public > interface that accepts and returns standard typing types, and internally > converts to our (nowhere else exposed) typehints types, performs the logic, > and converts back. Each of these in turn can then be updated, as needed and > orthogonally, to operate on the typing types natively (possibly via deference > to a third-party library). > > I think coder inference could be easily adopted to use typing types directly, > but it may be a fourth place where we do internal conversion first. Another > gotcha is special care may need to be taken if we ever need to pickle these > types (which IIRC may have issues). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-7049) Merge multiple input to one BeamUnionRel
[ https://issues.apache.org/jira/browse/BEAM-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928954#comment-16928954 ] Rui Wang commented on BEAM-7049: [~sridharG] I will do some testing on my side to check UnionMergeRule > Merge multiple input to one BeamUnionRel > > > Key: BEAM-7049 > URL: https://issues.apache.org/jira/browse/BEAM-7049 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: sridhar Reddy >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > BeamUnionRel assumes inputs are two and rejects more. So `a UNION b UNION c` > will have to be created as UNION(a, UNION(b, c)) and have two shuffles. If > BeamUnionRel can handle multiple shuffles, we will have only one shuffle -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (BEAM-8225) Skip portable runner tests when Docker is not installed.
Valentyn Tymofieiev created BEAM-8225: - Summary: Skip portable runner tests when Docker is not installed. Key: BEAM-8225 URL: https://issues.apache.org/jira/browse/BEAM-8225 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Valentyn Tymofieiev Assignee: Hannah Jiang -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8225) Skip portable runner tests when Docker is not installed.
[ https://issues.apache.org/jira/browse/BEAM-8225?focusedWorklogId=311828&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311828 ] ASF GitHub Bot logged work on BEAM-8225: Author: ASF GitHub Bot Created on: 13/Sep/19 00:05 Start Date: 13/Sep/19 00:05 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9557: [BEAM-8225] Portable runner test Bugfix URL: https://github.com/apache/beam/pull/9557 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311828) Remaining Estimate: 0h Time Spent: 10m > Skip portable runner tests when Docker is not installed. > - > > Key: BEAM-8225 > URL: https://issues.apache.org/jira/browse/BEAM-8225 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Hannah Jiang >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (BEAM-8224) The way we inspect arguments in Python does not work for lambda *_: ...
[ https://issues.apache.org/jira/browse/BEAM-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay updated BEAM-8224: -- Priority: Blocker (was: Major) > The way we inspect arguments in Python does not work for lambda *_: ... > --- > > Key: BEAM-8224 > URL: https://issues.apache.org/jira/browse/BEAM-8224 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Yueyang Qiu >Assignee: Udi Meiri >Priority: Blocker > Fix For: 2.16.0 > > > We need to make it work and create a test for this case. > > Assigning to Udi who authored > [https://github.com/apache/beam/pull/9283|https://www.google.com/url?q=https://github.com/apache/beam/pull/9283&sa=D&usg=AFQjCNETtLzhe9jPfGzLHJb2N4Sle-FuNw]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (BEAM-8224) The way we inspect arguments in Python does not work for lambda *_: ...
[ https://issues.apache.org/jira/browse/BEAM-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yueyang Qiu updated BEAM-8224: -- Summary: The way we inspect arguments in Python does not work for lambda *_: ... (was: The way we inspect arguments does not work for lambda *_: ...) > The way we inspect arguments in Python does not work for lambda *_: ... > --- > > Key: BEAM-8224 > URL: https://issues.apache.org/jira/browse/BEAM-8224 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Yueyang Qiu >Assignee: Udi Meiri >Priority: Major > Fix For: 2.16.0 > > > We need to make it work and create a test for this case. > > Assigning to Udi who authored > [https://github.com/apache/beam/pull/9283|https://www.google.com/url?q=https://github.com/apache/beam/pull/9283&sa=D&usg=AFQjCNETtLzhe9jPfGzLHJb2N4Sle-FuNw]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (BEAM-8224) The way we inspect arguments does not work for lambda *_: ...
Yueyang Qiu created BEAM-8224: - Summary: The way we inspect arguments does not work for lambda *_: ... Key: BEAM-8224 URL: https://issues.apache.org/jira/browse/BEAM-8224 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Yueyang Qiu Assignee: Udi Meiri Fix For: 2.16.0 We need to make it work and create a test for this case. Assigning to Udi who authored [https://github.com/apache/beam/pull/9283|https://www.google.com/url?q=https://github.com/apache/beam/pull/9283&sa=D&usg=AFQjCNETtLzhe9jPfGzLHJb2N4Sle-FuNw]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8222) Consider making insertId optional in BigQuery.insertAll
[ https://issues.apache.org/jira/browse/BEAM-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928946#comment-16928946 ] Nahuel Lofeudo commented on BEAM-8222: -- The request is to not populate the insertId field when calling insertAll(), in order to use BigQuery's Streaming API V2 as described here: [https://cloud.google.com/bigquery/quotas#streaming_inserts] " > Consider making insertId optional in BigQuery.insertAll > --- > > Key: BEAM-8222 > URL: https://issues.apache.org/jira/browse/BEAM-8222 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Boyuan Zhang >Priority: Major > > Current implementation of > StreamingWriteFn(https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteFn.java#L102) > sets insertId from input element, which is added an uniqueId by > https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java#L53. > Users report that if leaving insertId as empty, writing will be extremely > speeded up. Can we add an bqOption like, nonInsertId and emit empty id based > on this option? -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (BEAM-8223) Version number need to be updated in gradle.properties during release
[ https://issues.apache.org/jira/browse/BEAM-8223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Liu updated BEAM-8223: --- Fix Version/s: 2.16.0 > Version number need to be updated in gradle.properties during release > - > > Key: BEAM-8223 > URL: https://issues.apache.org/jira/browse/BEAM-8223 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Blocker > Fix For: 2.16.0 > > > New release should remove dev suffix from version number in gradle.properties. > Instead we use version=2.16.0-SNAPSHOT and python_sdk_version=2.16.0.dev in > master, we should change to version=2.16.0 and python_sdk_version=2.16.0 > during new release. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (BEAM-8223) Version number need to be updated in gradle.properties during release
Mark Liu created BEAM-8223: -- Summary: Version number need to be updated in gradle.properties during release Key: BEAM-8223 URL: https://issues.apache.org/jira/browse/BEAM-8223 Project: Beam Issue Type: Bug Components: build-system Reporter: Mark Liu Assignee: Mark Liu New release should remove dev suffix from version number in gradle.properties. Instead we use version=2.16.0-SNAPSHOT and python_sdk_version=2.16.0.dev in master, we should change to version=2.16.0 and python_sdk_version=2.16.0 during new release. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8215) Wordcount 1GB Python PKB benchmarks sometimes fail with uninformative error
[
https://issues.apache.org/jira/browse/BEAM-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928929#comment-16928929
]
Valentyn Tymofieiev commented on BEAM-8215:
---
To clarify, these performance tests run as postcommits, but they are different
from Python 3 integration test postcommits.
> Wordcount 1GB Python PKB benchmarks sometimes fail with uninformative error
> ---
>
> Key: BEAM-8215
> URL: https://issues.apache.org/jira/browse/BEAM-8215
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Mark Liu
>Priority: Major
>
> Example:
> https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/452/console
> {noformat}
> 12:09:27 2019-09-11 19:09:27,655 a47400ce MainThread
> beam_integration_benchmark(1/1) ERRORError during benchmark
> beam_integration_benchmark
> 12:09:27 Traceback (most recent call last):
> 12:09:27 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
> line 841, in RunBenchmark
> 12:09:27 DoRunPhase(spec, collector, detailed_timer)
> 12:09:27 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
> line 687, in DoRunPhase
> 12:09:27 samples = spec.BenchmarkRun(spec)
> 12:09:27 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
> line 160, in Run
> 12:09:27 job_type=job_type)
> 12:09:27 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
> line 91, in SubmitJob
> 12:09:27 assert retcode == 0, "Integration Test Failed."
> 12:09:27 AssertionError: Integration Test Failed.
> {noformat}
> It seems like Job submission failed, but there are no details. I talked with
> [~markflyhigh], and sounds like we plan to stop using PKB in favor of another
> framework.
> Assigning to Mark for now to triage follow up or reassign as appropriate.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7060) Design Py3-compatible typehints annotation support in Beam 3.
[ https://issues.apache.org/jira/browse/BEAM-7060?focusedWorklogId=311803&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311803 ] ASF GitHub Bot logged work on BEAM-7060: Author: ASF GitHub Bot Created on: 12/Sep/19 22:57 Start Date: 12/Sep/19 22:57 Worklog Time Spent: 10m Work Description: udim commented on pull request #9562: [BEAM-7060] Prefer inspect module over funcsigs URL: https://github.com/apache/beam/pull/9562 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311803) Time Spent: 16h (was: 15h 50m) > Design Py3-compatible typehints annotation support in Beam 3. > - > > Key: BEAM-7060 > URL: https://issues.apache.org/jira/browse/BEAM-7060 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Udi Meiri >Priority: Major > Fix For: 2.16.0 > > Time Spent: 16h > Remaining Estimate: 0h > > Existing [Typehints implementaiton in > Beam|[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/ > ] heavily relies on internal details of CPython implementation, and some of > the assumptions of this implementation broke as of Python 3.6, see for > example: https://issues.apache.org/jira/browse/BEAM-6877, which makes > typehints support unusable on Python 3.6 as of now. [Python 3 Kanban > Board|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail] > lists several specific typehints-related breakages, prefixed with "TypeHints > Py3 Error". > We need to decide whether to: > - Deprecate in-house typehints implementation. > - Continue to support in-house implementation, which at this point is a stale > code and has other known issues. > - Attempt to use some off-the-shelf libraries for supporting > type-annotations, like Pytype, Mypy, PyAnnotate. > WRT to this decision we also need to plan on immediate next steps to unblock > adoption of Beam for Python 3.6+ users. One potential option may be to have > Beam SDK ignore any typehint annotations on Py 3.6+. > cc: [~udim], [~altay], [~robertwb]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (BEAM-8222) Consider making insertId optional in BigQuery.insertAll
Boyuan Zhang created BEAM-8222: -- Summary: Consider making insertId optional in BigQuery.insertAll Key: BEAM-8222 URL: https://issues.apache.org/jira/browse/BEAM-8222 Project: Beam Issue Type: New Feature Components: io-java-gcp Reporter: Boyuan Zhang Current implementation of StreamingWriteFn(https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteFn.java#L102) sets insertId from input element, which is added an uniqueId by https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java#L53. Users report that if leaving insertId as empty, writing will be extremely speeded up. Can we add an bqOption like, nonInsertId and emit empty id based on this option? -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311797&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311797 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 12/Sep/19 22:49 Start Date: 12/Sep/19 22:49 Worklog Time Spent: 10m Work Description: vectorijk commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-531035981 Thanks @amaliujia! I'll squash commits later This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311797) Time Spent: 9h 10m (was: 9h) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 9h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8111) SchemaCoder broken on DataflowRunner
[
https://issues.apache.org/jira/browse/BEAM-8111?focusedWorklogId=311795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311795
]
ASF GitHub Bot logged work on BEAM-8111:
Author: ASF GitHub Bot
Created on: 12/Sep/19 22:45
Start Date: 12/Sep/19 22:45
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #9446:
[BEAM-8111] Enable CloudObjectsTest$DefaultCoders
URL: https://github.com/apache/beam/pull/9446#discussion_r323979264
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
##
@@ -100,4 +99,47 @@ public boolean consistentWithEquals() {
public String toString() {
return "SchemaCoder: " + rowCoder.toString();
}
+
+ @Override
+ public boolean equals(Object o) {
+if (this == o) {
+ return true;
+}
+if (o == null || getClass() != o.getClass()) {
+ return false;
+}
+SchemaCoder that = (SchemaCoder) o;
+return rowCoder.equals(that.rowCoder)
+&& toRowFunction.equals(that.toRowFunction)
+&& fromRowFunction.equals(that.fromRowFunction);
Review comment:
Another alternative could be to add something like
`assertEquivalentSchemaCoder` that just checks schema and type, rather than
continuing down this rabbit hole.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311795)
Time Spent: 3h 40m (was: 3.5h)
> SchemaCoder broken on DataflowRunner
>
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
> broke SchemaCoder on Dataflow. When translating a schema that uses logical
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail
> on Dataflow in 2.15.0
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-8217) py3 sdk harness status server does not print thread stack
[
https://issues.apache.org/jira/browse/BEAM-8217?focusedWorklogId=311794&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311794
]
ASF GitHub Bot logged work on BEAM-8217:
Author: ASF GitHub Bot
Created on: 12/Sep/19 22:42
Start Date: 12/Sep/19 22:42
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #9559: [BEAM-8217]
Cherry-pick PR #9551: Encode str to bytes py3 in sdk harness status server
URL: https://github.com/apache/beam/pull/9559#issuecomment-531034411
@Hannah-Jiang is working on a fix that address the Portable_python failure.
Not sure if Precommit has same root cause.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311794)
Time Spent: 1h 20m (was: 1h 10m)
> py3 sdk harness status server does not print thread stack
> -
>
> Key: BEAM-8217
> URL: https://issues.apache.org/jira/browse/BEAM-8217
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Ankur Goenka
>Priority: Blocker
> Fix For: 2.16.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The calls are failing with
>
> Exception happened during processing of request from ('127.0.0.1', 56106)
> Traceback (most recent call last):
> File "/usr/lib/python3.6/socketserver.py", line 320, in
> _handle_request_noblock
> self.process_request(request, client_address)
> File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
> self.finish_request(request, client_address)
> File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
> self.RequestHandlerClass(request, client_address, self)
> File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
> self.handle()
> File "/usr/lib/python3.6/http/server.py", line 418, in handle
> self.handle_one_request()
> File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
> method()
> File "a.py", line 70, in do_GET
> self.wfile.write(line)
> File "/usr/lib/python3.6/socketserver.py", line 803, in write
> self._sock.sendall(b)
> TypeError: a bytes-like object is required, not 'str'
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-876) Support schemaUpdateOption in BigQueryIO
[ https://issues.apache.org/jira/browse/BEAM-876?focusedWorklogId=311790&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311790 ] ASF GitHub Bot logged work on BEAM-876: --- Author: ASF GitHub Bot Created on: 12/Sep/19 22:32 Start Date: 12/Sep/19 22:32 Worklog Time Spent: 10m Work Description: ziel commented on issue #9524: [BEAM-876] Support schemaUpdateOption in BigQueryIO URL: https://github.com/apache/beam/pull/9524#issuecomment-529585211 R: @lukecwik @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311790) Time Spent: 0.5h (was: 20m) > Support schemaUpdateOption in BigQueryIO > > > Key: BEAM-876 > URL: https://issues.apache.org/jira/browse/BEAM-876 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Eugene Kirpichov >Assignee: canaan silberberg >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > BigQuery recently added support for updating the schema as a side effect of > the load job. > Here is the relevant API method in JobConfigurationLoad: > https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/JobConfigurationLoad.html#setSchemaUpdateOptions(java.util.List) > BigQueryIO should support this too. See user request for this: > http://stackoverflow.com/questions/40333245/is-it-possible-to-update-schema-while-doing-a-load-into-an-existing-bigquery-tab -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual
[
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=311784&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311784
]
ASF GitHub Bot logged work on BEAM-8105:
Author: ASF GitHub Bot
Created on: 12/Sep/19 22:25
Start Date: 12/Sep/19 22:25
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on pull request #9510:
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r323974093
##
File path: website/src/contribute/release-guide.md
##
@@ -305,6 +332,7 @@ There are 2 ways to cut a release branch: either running
automation script(recom
DEV=${RELEASE}.dev
sed -i -e "s/${DEV}/${RELEASE}/g" sdks/python/apache_beam/version.py
+ sed -i -e "s/${DEV}/${RELEASE}/g" gradle.properties
Review comment:
great catch, added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311784)
Time Spent: 7.5h (was: 7h 20m)
> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
> Issue Type: Sub-task
> Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (BEAM-8221) NullPointerException in reading from non-existent Kafka topic
Chandan Kumar created BEAM-8221:
---
Summary: NullPointerException in reading from non-existent Kafka
topic
Key: BEAM-8221
URL: https://issues.apache.org/jira/browse/BEAM-8221
Project: Beam
Issue Type: Bug
Components: io-java-kafka
Reporter: Chandan Kumar
Get this exception when launch apache beam job, which tries to read from
non-existent Kafka topic:
{code:java}
Exception in thread "main" java.lang.RuntimeException:
java.lang.NullPointerException
at
org.apache.beam.runners.dataflow.ReadTranslator.translateReadHelper(ReadTranslator.java:54)
at
org.apache.beam.runners.dataflow.DataflowRunner$StreamingUnboundedRead$ReadWithIdsTranslator.translate(DataflowRunner.java:1508)
at
org.apache.beam.runners.dataflow.DataflowRunner$StreamingUnboundedRead$ReadWithIdsTranslator.translate(DataflowRunner.java:1505)
at
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:475)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
at
org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
at
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:414)
at
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:173)
at
org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:737)
at
org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:179)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
at
com.paloaltonetworks.cortex.replication.ingest.ReplicationIngestor.run(ReplicationIngestor.java:109)
at
com.paloaltonetworks.cortex.replication.ingest.ReplicationIngestor.main(ReplicationIngestor.java:64)
Caused by: java.lang.NullPointerException
at
org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.split(KafkaUnboundedSource.java:66)
at
org.apache.beam.runners.dataflow.internal.CustomSources.serializeToCloudSource(CustomSources.java:87)
at
org.apache.beam.runners.dataflow.ReadTranslator.translateReadHelper(ReadTranslator.java:51)
... 18 more {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual
[ https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=311777&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311777 ] ASF GitHub Bot logged work on BEAM-8105: Author: ASF GitHub Bot Created on: 12/Sep/19 22:02 Start Date: 12/Sep/19 22:02 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #9510: [BEAM-8105] update release guide with docker images URL: https://github.com/apache/beam/pull/9510#issuecomment-531024250 Can you also update this section `Change and commit version number in release branch:` to include `gradle.properties`? This is how release manager verify changes after run cut_release_branch.sh This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311777) Time Spent: 7h 20m (was: 7h 10m) > Add container publishing instruction to release manual > -- > > Key: BEAM-8105 > URL: https://issues.apache.org/jira/browse/BEAM-8105 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.16.0 > > Time Spent: 7h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual
[ https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=311776&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311776 ] ASF GitHub Bot logged work on BEAM-8105: Author: ASF GitHub Bot Created on: 12/Sep/19 22:01 Start Date: 12/Sep/19 22:01 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #9510: [BEAM-8105] update release guide with docker images URL: https://github.com/apache/beam/pull/9510#issuecomment-531024250 Can you also update this section `Change and commit version number in release branch:` to include `gradle.properties`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311776) Time Spent: 7h 10m (was: 7h) > Add container publishing instruction to release manual > -- > > Key: BEAM-8105 > URL: https://issues.apache.org/jira/browse/BEAM-8105 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.16.0 > > Time Spent: 7h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311772&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311772 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 12/Sep/19 21:57 Start Date: 12/Sep/19 21:57 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-531022889 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311772) Time Spent: 8h 50m (was: 8h 40m) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311773&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311773 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 12/Sep/19 21:57 Start Date: 12/Sep/19 21:57 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-531022927 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311773) Time Spent: 9h (was: 8h 50m) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 9h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8111) SchemaCoder broken on DataflowRunner
[
https://issues.apache.org/jira/browse/BEAM-8111?focusedWorklogId=311774&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311774
]
ASF GitHub Bot logged work on BEAM-8111:
Author: ASF GitHub Bot
Created on: 12/Sep/19 21:57
Start Date: 12/Sep/19 21:57
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #9446:
[BEAM-8111] Enable CloudObjectsTest$DefaultCoders
URL: https://github.com/apache/beam/pull/9446#discussion_r323966059
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
##
@@ -100,4 +99,47 @@ public boolean consistentWithEquals() {
public String toString() {
return "SchemaCoder: " + rowCoder.toString();
}
+
+ @Override
+ public boolean equals(Object o) {
+if (this == o) {
+ return true;
+}
+if (o == null || getClass() != o.getClass()) {
+ return false;
+}
+SchemaCoder that = (SchemaCoder) o;
+return rowCoder.equals(that.rowCoder)
+&& toRowFunction.equals(that.toRowFunction)
+&& fromRowFunction.equals(that.fromRowFunction);
Review comment:
That's true. I was thinking it's not such a big deal to get false negatives
when lambdas are used, since I really just want the equality check to use in
tests.
What do you think about updating the various schema providers to create
Function sub-classes (with equals implemented) instead of using lambdas?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311774)
Time Spent: 3.5h (was: 3h 20m)
> SchemaCoder broken on DataflowRunner
>
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
> broke SchemaCoder on Dataflow. When translating a schema that uses logical
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail
> on Dataflow in 2.15.0
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual
[
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=311770&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311770
]
ASF GitHub Bot logged work on BEAM-8105:
Author: ASF GitHub Bot
Created on: 12/Sep/19 21:49
Start Date: 12/Sep/19 21:49
Worklog Time Spent: 10m
Work Description: markflyhigh commented on pull request #9510:
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r323963515
##
File path: website/src/contribute/release-guide.md
##
@@ -305,6 +332,7 @@ There are 2 ways to cut a release branch: either running
automation script(recom
DEV=${RELEASE}.dev
sed -i -e "s/${DEV}/${RELEASE}/g" sdks/python/apache_beam/version.py
+ sed -i -e "s/${DEV}/${RELEASE}/g" gradle.properties
Review comment:
should cut_release_branch.sh also need to be updated?
[Here](https://github.com/apache/beam/blob/master/release/src/main/scripts/cut_release_branch.sh#L127)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311770)
Time Spent: 7h (was: 6h 50m)
> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
> Issue Type: Sub-task
> Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work started] (BEAM-8098) BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK correspond
[ https://issues.apache.org/jira/browse/BEAM-8098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-8098 started by Yueyang Qiu. - > BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK > correspond > --- > > Key: BEAM-8098 > URL: https://issues.apache.org/jira/browse/BEAM-8098 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > While working on [https://github.com/apache/beam/pull/9144], I realized there > is a gap in BigQueryIO documentation on mapping between data types defined in > BigQuery and in Beam SDK. > > For example, if a user reads a BYTES field from BigQuery into Beam, it will > be represented as java.nio.ByteBuffer type in Beam Java SDK. The user will > need to do an explicit type cast to ByteBuffer in order to use the data, but > there is no easy way the user can know which type they should cast to, unless > digging into BigQueryIO's implementation (Java - Avro - BigQuery). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Closed] (BEAM-8098) BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK correspond
[ https://issues.apache.org/jira/browse/BEAM-8098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yueyang Qiu closed BEAM-8098. - Fix Version/s: 2.17.0 Resolution: Fixed > BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK > correspond > --- > > Key: BEAM-8098 > URL: https://issues.apache.org/jira/browse/BEAM-8098 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 40m > Remaining Estimate: 0h > > While working on [https://github.com/apache/beam/pull/9144], I realized there > is a gap in BigQueryIO documentation on mapping between data types defined in > BigQuery and in Beam SDK. > > For example, if a user reads a BYTES field from BigQuery into Beam, it will > be represented as java.nio.ByteBuffer type in Beam Java SDK. The user will > need to do an explicit type cast to ByteBuffer in order to use the data, but > there is no easy way the user can know which type they should cast to, unless > digging into BigQueryIO's implementation (Java - Avro - BigQuery). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8098) BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK correspond
[ https://issues.apache.org/jira/browse/BEAM-8098?focusedWorklogId=311766&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311766 ] ASF GitHub Bot logged work on BEAM-8098: Author: ASF GitHub Bot Created on: 12/Sep/19 21:33 Start Date: 12/Sep/19 21:33 Worklog Time Spent: 10m Work Description: pabloem commented on issue #9555: [BEAM-8098] Improve documentation on BigQueryIO URL: https://github.com/apache/beam/pull/9555#issuecomment-531015422 Thanks Robin! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311766) Time Spent: 0.5h (was: 20m) > BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK > correspond > --- > > Key: BEAM-8098 > URL: https://issues.apache.org/jira/browse/BEAM-8098 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > While working on [https://github.com/apache/beam/pull/9144], I realized there > is a gap in BigQueryIO documentation on mapping between data types defined in > BigQuery and in Beam SDK. > > For example, if a user reads a BYTES field from BigQuery into Beam, it will > be represented as java.nio.ByteBuffer type in Beam Java SDK. The user will > need to do an explicit type cast to ByteBuffer in order to use the data, but > there is no easy way the user can know which type they should cast to, unless > digging into BigQueryIO's implementation (Java - Avro - BigQuery). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8098) BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK correspond
[ https://issues.apache.org/jira/browse/BEAM-8098?focusedWorklogId=311767&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311767 ] ASF GitHub Bot logged work on BEAM-8098: Author: ASF GitHub Bot Created on: 12/Sep/19 21:33 Start Date: 12/Sep/19 21:33 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #9555: [BEAM-8098] Improve documentation on BigQueryIO URL: https://github.com/apache/beam/pull/9555 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311767) Time Spent: 40m (was: 0.5h) > BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK > correspond > --- > > Key: BEAM-8098 > URL: https://issues.apache.org/jira/browse/BEAM-8098 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > While working on [https://github.com/apache/beam/pull/9144], I realized there > is a gap in BigQueryIO documentation on mapping between data types defined in > BigQuery and in Beam SDK. > > For example, if a user reads a BYTES field from BigQuery into Beam, it will > be represented as java.nio.ByteBuffer type in Beam Java SDK. The user will > need to do an explicit type cast to ByteBuffer in order to use the data, but > there is no easy way the user can know which type they should cast to, unless > digging into BigQueryIO's implementation (Java - Avro - BigQuery). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7060) Design Py3-compatible typehints annotation support in Beam 3.
[
https://issues.apache.org/jira/browse/BEAM-7060?focusedWorklogId=311764&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311764
]
ASF GitHub Bot logged work on BEAM-7060:
Author: ASF GitHub Bot
Created on: 12/Sep/19 21:32
Start Date: 12/Sep/19 21:32
Worklog Time Spent: 10m
Work Description: udim commented on pull request #9562: [BEAM-7060]
Prefer inspect module over funcsigs
URL: https://github.com/apache/beam/pull/9562
In Python 3 with funcsigs installed, prefer inspect.{signature,Paramter}
over funcsigs equivalents.
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] [**Choose
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
Post-Commit Tests Status (on master branch)
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](h
[jira] [Work logged] (BEAM-7060) Design Py3-compatible typehints annotation support in Beam 3.
[ https://issues.apache.org/jira/browse/BEAM-7060?focusedWorklogId=311765&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311765 ] ASF GitHub Bot logged work on BEAM-7060: Author: ASF GitHub Bot Created on: 12/Sep/19 21:32 Start Date: 12/Sep/19 21:32 Worklog Time Spent: 10m Work Description: udim commented on issue #9562: [BEAM-7060] Prefer inspect module over funcsigs URL: https://github.com/apache/beam/pull/9562#issuecomment-531015227 R: @robinyqiu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311765) Time Spent: 15h 50m (was: 15h 40m) > Design Py3-compatible typehints annotation support in Beam 3. > - > > Key: BEAM-7060 > URL: https://issues.apache.org/jira/browse/BEAM-7060 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Udi Meiri >Priority: Major > Fix For: 2.16.0 > > Time Spent: 15h 50m > Remaining Estimate: 0h > > Existing [Typehints implementaiton in > Beam|[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/ > ] heavily relies on internal details of CPython implementation, and some of > the assumptions of this implementation broke as of Python 3.6, see for > example: https://issues.apache.org/jira/browse/BEAM-6877, which makes > typehints support unusable on Python 3.6 as of now. [Python 3 Kanban > Board|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail] > lists several specific typehints-related breakages, prefixed with "TypeHints > Py3 Error". > We need to decide whether to: > - Deprecate in-house typehints implementation. > - Continue to support in-house implementation, which at this point is a stale > code and has other known issues. > - Attempt to use some off-the-shelf libraries for supporting > type-annotations, like Pytype, Mypy, PyAnnotate. > WRT to this decision we also need to plan on immediate next steps to unblock > adoption of Beam for Python 3.6+ users. One potential option may be to have > Beam SDK ignore any typehint annotations on Py 3.6+. > cc: [~udim], [~altay], [~robertwb]. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[ https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=311753&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311753 ] ASF GitHub Bot logged work on BEAM-7389: Author: ASF GitHub Bot Created on: 12/Sep/19 21:20 Start Date: 12/Sep/19 21:20 Worklog Time Spent: 10m Work Description: davidcavazos commented on pull request #9561: [BEAM-7389] Generate notebook for filter URL: https://github.com/apache/beam/pull/9561#discussion_r323954208 ## File path: website/notebooks/generate.py ## @@ -0,0 +1,73 @@ +#!/usr/bin/env python +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# To run, you will have to install `md2ipynb`. Review comment: I'm adding a more thorough description. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311753) Time Spent: 59.5h (was: 59h 20m) > Colab examples for element-wise transforms (Python) > --- > > Key: BEAM-7389 > URL: https://issues.apache.org/jira/browse/BEAM-7389 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Rose Nguyen >Assignee: David Cavazos >Priority: Minor > Time Spent: 59.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[
https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=311751&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311751
]
ASF GitHub Bot logged work on BEAM-7389:
Author: ASF GitHub Bot
Created on: 12/Sep/19 21:16
Start Date: 12/Sep/19 21:16
Worklog Time Spent: 10m
Work Description: davidcavazos commented on pull request #9561:
[BEAM-7389] Generate notebook for filter
URL: https://github.com/apache/beam/pull/9561#discussion_r323952813
##
File path: website/src/documentation/transforms/python/element-wise/filter.md
##
@@ -78,18 +92,32 @@ We can also use lambda functions to simplify **Example 1**.
```py
{% github_sample
/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py
tag:filter_lambda %}```
+{:.notebook-skip}
Output `PCollection` after `Filter`:
+{:.notebook-skip}
```
{% github_sample
/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py
tag:perennials %}```
-
+{:.notebook-skip}
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py";>
+href="https://colab.research.google.com/github/{{ site.branch_repo
}}/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb">
Review comment:
It's a little hard to see in HTML, but it's actually a new button to run the
notebook, and we're keeping the one to see the source code in github.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311751)
Time Spent: 59h 20m (was: 59h 10m)
> Colab examples for element-wise transforms (Python)
> ---
>
> Key: BEAM-7389
> URL: https://issues.apache.org/jira/browse/BEAM-7389
> Project: Beam
> Issue Type: Improvement
> Components: website
>Reporter: Rose Nguyen
>Assignee: David Cavazos
>Priority: Minor
> Time Spent: 59h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[
https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=311748&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311748
]
ASF GitHub Bot logged work on BEAM-7389:
Author: ASF GitHub Bot
Created on: 12/Sep/19 21:14
Start Date: 12/Sep/19 21:14
Worklog Time Spent: 10m
Work Description: davidcavazos commented on pull request #9561:
[BEAM-7389] Generate notebook for filter
URL: https://github.com/apache/beam/pull/9561#discussion_r323952228
##
File path:
examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb
##
@@ -0,0 +1,521 @@
+{
Review comment:
You can look at it in Colab, I had the link in the comment above alongside
the staged docs.
https://colab.research.google.com/github/davidcavazos/beam/blob/notebook-filter/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311748)
Time Spent: 59h 10m (was: 59h)
> Colab examples for element-wise transforms (Python)
> ---
>
> Key: BEAM-7389
> URL: https://issues.apache.org/jira/browse/BEAM-7389
> Project: Beam
> Issue Type: Improvement
> Components: website
>Reporter: Rose Nguyen
>Assignee: David Cavazos
>Priority: Minor
> Time Spent: 59h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7911) test_corrupted_file test flaky
[
https://issues.apache.org/jira/browse/BEAM-7911?focusedWorklogId=311744&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311744
]
ASF GitHub Bot logged work on BEAM-7911:
Author: ASF GitHub Bot
Created on: 12/Sep/19 21:13
Start Date: 12/Sep/19 21:13
Worklog Time Spent: 10m
Work Description: udim commented on pull request #9554: [BEAM-7911] Fix
flakiness in avroio_test.py
URL: https://github.com/apache/beam/pull/9554
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311744)
Time Spent: 50m (was: 40m)
> test_corrupted_file test flaky
> --
>
> Key: BEAM-7911
> URL: https://issues.apache.org/jira/browse/BEAM-7911
> Project: Beam
> Issue Type: Bug
> Components: io-py-avro, test-failures
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Looks like a flake:
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7924/consoleFull
> 11:22:24
> 11:22:24
> ==
> 11:22:24 ERROR: test_corrupted_file (apache_beam.io.avroio_test.TestFastAvro)
> 11:22:24
> --
> 11:22:24 Traceback (most recent call last):
> 11:22:24 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/avroio_test.py",
> line 380, in test_corrupted_file
> 11:22:24 self.assertEqual(0, exn.exception.message.find('Unexpected sync
> marker'))
> 11:22:24 AttributeError: '_AssertRaisesContext' object has no attribute
> 'exception'
> 11:22:24 >> begin captured logging <<
>
> 11:22:24 apache_beam.io.filesystem: DEBUG: translate_pattern:
> '/tmp/tmpckgw8mk6' -> '\\/tmp\\/tmpckgw8mk6'
> 11:22:24 - >> end captured logging <<
> -
> 11:22:24
> 11:22:24
> --
> 11:22:24 XML:
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/nosetests.xml
> 11:22:24
> --
> 11:22:24 Ran 2455 tests in 1297.267s
> 11:22:24
> 11:22:24 FAILED (SKIP=546, errors=1)
> 11:22:24 ERROR: InvocationError for command
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36/py36/bin/python
> setup.py nosetests (exited with code 1)
> 11:22:24 py36 run-test-post: commands[0] |
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
> 11:22:24 ___ summary
>
> 11:22:24 ERROR: py36: commands failed
> 11:22:24
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[ https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=311743&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311743 ] ASF GitHub Bot logged work on BEAM-7389: Author: ASF GitHub Bot Created on: 12/Sep/19 21:13 Start Date: 12/Sep/19 21:13 Worklog Time Spent: 10m Work Description: davidcavazos commented on pull request #9561: [BEAM-7389] Generate notebook for filter URL: https://github.com/apache/beam/pull/9561#discussion_r323951671 ## File path: website/notebooks/docs.yaml ## @@ -0,0 +1,108 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Python transform catalog +documentation/transforms/python/element-wise/filter: + title: Filter - element-wise transform + languages: py + imports: +0: [license.md] Review comment: Yes, they are common things for all files. They import the license and setup steps (install `apache-beam`), which are not part of the Markdown files, but are needed in the notebook. I was thinking on always importing these by default, but if we need a different setup like installing `apache-beam[gcp]` we would have to define it here. I thought it was more explicit to have them defined individually. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311743) Time Spent: 59h (was: 58h 50m) > Colab examples for element-wise transforms (Python) > --- > > Key: BEAM-7389 > URL: https://issues.apache.org/jira/browse/BEAM-7389 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Rose Nguyen >Assignee: David Cavazos >Priority: Minor > Time Spent: 59h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8217) py3 sdk harness status server does not print thread stack
[
https://issues.apache.org/jira/browse/BEAM-8217?focusedWorklogId=311739&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311739
]
ASF GitHub Bot logged work on BEAM-8217:
Author: ASF GitHub Bot
Created on: 12/Sep/19 21:10
Start Date: 12/Sep/19 21:10
Worklog Time Spent: 10m
Work Description: angoenka commented on issue #9559: [BEAM-8217]
Cherry-pick PR #9551: Encode str to bytes py3 in sdk harness status server
URL: https://github.com/apache/beam/pull/9559#issuecomment-531008255
@markflyhigh python precommit are failing while pulling the docker image.
Are you aware of such problem?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311739)
Time Spent: 1h 10m (was: 1h)
> py3 sdk harness status server does not print thread stack
> -
>
> Key: BEAM-8217
> URL: https://issues.apache.org/jira/browse/BEAM-8217
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Ankur Goenka
>Priority: Blocker
> Fix For: 2.16.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The calls are failing with
>
> Exception happened during processing of request from ('127.0.0.1', 56106)
> Traceback (most recent call last):
> File "/usr/lib/python3.6/socketserver.py", line 320, in
> _handle_request_noblock
> self.process_request(request, client_address)
> File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
> self.finish_request(request, client_address)
> File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
> self.RequestHandlerClass(request, client_address, self)
> File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
> self.handle()
> File "/usr/lib/python3.6/http/server.py", line 418, in handle
> self.handle_one_request()
> File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
> method()
> File "a.py", line 70, in do_GET
> self.wfile.write(line)
> File "/usr/lib/python3.6/socketserver.py", line 803, in write
> self._sock.sendall(b)
> TypeError: a bytes-like object is required, not 'str'
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[
https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=311735&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311735
]
ASF GitHub Bot logged work on BEAM-7389:
Author: ASF GitHub Bot
Created on: 12/Sep/19 21:08
Start Date: 12/Sep/19 21:08
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9561: [BEAM-7389]
Generate notebook for filter
URL: https://github.com/apache/beam/pull/9561#discussion_r323950004
##
File path:
examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb
##
@@ -0,0 +1,521 @@
+{
Review comment:
Hard to review this file really. I looked using the UI that displays
notebooks inside github. It looked ok but there were also issues, like missing
images.
How can I view this correctly and review it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311735)
Time Spent: 58.5h (was: 58h 20m)
> Colab examples for element-wise transforms (Python)
> ---
>
> Key: BEAM-7389
> URL: https://issues.apache.org/jira/browse/BEAM-7389
> Project: Beam
> Issue Type: Improvement
> Components: website
>Reporter: Rose Nguyen
>Assignee: David Cavazos
>Priority: Minor
> Time Spent: 58.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[ https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=311736&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311736 ] ASF GitHub Bot logged work on BEAM-7389: Author: ASF GitHub Bot Created on: 12/Sep/19 21:08 Start Date: 12/Sep/19 21:08 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9561: [BEAM-7389] Generate notebook for filter URL: https://github.com/apache/beam/pull/9561#discussion_r323949219 ## File path: website/notebooks/generate.py ## @@ -0,0 +1,73 @@ +#!/usr/bin/env python +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# To run, you will have to install `md2ipynb`. Review comment: You probably need to add a more comprehensive help message about why would people run this, how would they run it, what do they need to have already installed to run it ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311736) Time Spent: 58h 40m (was: 58.5h) > Colab examples for element-wise transforms (Python) > --- > > Key: BEAM-7389 > URL: https://issues.apache.org/jira/browse/BEAM-7389 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Rose Nguyen >Assignee: David Cavazos >Priority: Minor > Time Spent: 58h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[
https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=311737&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311737
]
ASF GitHub Bot logged work on BEAM-7389:
Author: ASF GitHub Bot
Created on: 12/Sep/19 21:08
Start Date: 12/Sep/19 21:08
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9561: [BEAM-7389]
Generate notebook for filter
URL: https://github.com/apache/beam/pull/9561#discussion_r323949505
##
File path: website/src/documentation/transforms/python/element-wise/filter.md
##
@@ -78,18 +92,32 @@ We can also use lambda functions to simplify **Example 1**.
```py
{% github_sample
/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py
tag:filter_lambda %}```
+{:.notebook-skip}
Output `PCollection` after `Filter`:
+{:.notebook-skip}
```
{% github_sample
/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py
tag:perennials %}```
-
+{:.notebook-skip}
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py";>
+href="https://colab.research.google.com/github/{{ site.branch_repo
}}/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb">
Review comment:
We just recently added these snippets. Do we no longer need them? Is
notebooks replacing them?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311737)
Time Spent: 58h 50m (was: 58h 40m)
> Colab examples for element-wise transforms (Python)
> ---
>
> Key: BEAM-7389
> URL: https://issues.apache.org/jira/browse/BEAM-7389
> Project: Beam
> Issue Type: Improvement
> Components: website
>Reporter: Rose Nguyen
>Assignee: David Cavazos
>Priority: Minor
> Time Spent: 58h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[ https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=311734&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311734 ] ASF GitHub Bot logged work on BEAM-7389: Author: ASF GitHub Bot Created on: 12/Sep/19 21:08 Start Date: 12/Sep/19 21:08 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9561: [BEAM-7389] Generate notebook for filter URL: https://github.com/apache/beam/pull/9561#discussion_r323948933 ## File path: website/notebooks/docs.yaml ## @@ -0,0 +1,108 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Python transform catalog +documentation/transforms/python/element-wise/filter: + title: Filter - element-wise transform + languages: py + imports: +0: [license.md] Review comment: What does it mean that it imports these files? What is the purpose of these files? Are they just common things to all files? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311734) Time Spent: 58h 20m (was: 58h 10m) > Colab examples for element-wise transforms (Python) > --- > > Key: BEAM-7389 > URL: https://issues.apache.org/jira/browse/BEAM-7389 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Rose Nguyen >Assignee: David Cavazos >Priority: Minor > Time Spent: 58h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8111) SchemaCoder broken on DataflowRunner
[
https://issues.apache.org/jira/browse/BEAM-8111?focusedWorklogId=311718&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311718
]
ASF GitHub Bot logged work on BEAM-8111:
Author: ASF GitHub Bot
Created on: 12/Sep/19 20:55
Start Date: 12/Sep/19 20:55
Worklog Time Spent: 10m
Work Description: reuvenlax commented on pull request #9446: [BEAM-8111]
Enable CloudObjectsTest$DefaultCoders
URL: https://github.com/apache/beam/pull/9446#discussion_r323945107
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
##
@@ -100,4 +99,47 @@ public boolean consistentWithEquals() {
public String toString() {
return "SchemaCoder: " + rowCoder.toString();
}
+
+ @Override
+ public boolean equals(Object o) {
+if (this == o) {
+ return true;
+}
+if (o == null || getClass() != o.getClass()) {
+ return false;
+}
+SchemaCoder that = (SchemaCoder) o;
+return rowCoder.equals(that.rowCoder)
+&& toRowFunction.equals(that.toRowFunction)
+&& fromRowFunction.equals(that.fromRowFunction);
Review comment:
Sounds good in theory. In practice these functions are usually lambdas, so
we might have trouble making this work.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311718)
Time Spent: 3h 20m (was: 3h 10m)
> SchemaCoder broken on DataflowRunner
>
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
> broke SchemaCoder on Dataflow. When translating a schema that uses logical
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail
> on Dataflow in 2.15.0
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-8164) Correct document for building the python SDK harness container
[
https://issues.apache.org/jira/browse/BEAM-8164?focusedWorklogId=311717&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311717
]
ASF GitHub Bot logged work on BEAM-8164:
Author: ASF GitHub Bot
Created on: 12/Sep/19 20:54
Start Date: 12/Sep/19 20:54
Worklog Time Spent: 10m
Work Description: ibzib commented on issue #9536: [BEAM-8164][website]
Correct document for building the python SDK ha…
URL: https://github.com/apache/beam/pull/9536#issuecomment-531002621
Hi Jincheng, sorry I did not see this PR before filing my own that partially
addressed some of these issues. A few thoughts:
- Removing the outdated instructions LGTM.
- We should use python 3 in all the relevant examples, as py2 will be
deprecated soon.
- In a couple of the runner examples, we are instructing users to use
loopback mode, which means building the docker containers is actually not
needed (I missed this in #9545 as well). So we can just remove that step.
Wherever we actually do use the docker environment (the default), we should
probably change it to loopback, because loopback mode is easier to use for
newcomers to Beam.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311717)
Time Spent: 40m (was: 0.5h)
> Correct document for building the python SDK harness container
> --
>
> Key: BEAM-8164
> URL: https://issues.apache.org/jira/browse/BEAM-8164
> Project: Beam
> Issue Type: Bug
> Components: website
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In the runner document, it is described that we can use the command:
> `./gradlew :sdks:python:container:docker`
> to Build the SDK harness container, see
> ([https://beam.apache.org/documentation/runners/flink/)].
> However, the docker config has been removed with the latest python3 docker
> related commit [1] the command would failed with the following error message.
> {code:java}
> > Task :sdks:python:container:docker FAILED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':sdks:python:container:docker'.
> > name is a required docker configuration item.{code}
> I think we should also adapt the document with command: `./gradlew
> :sdks:python:container:py2:docker`? Or add the config when run
> `:sdks:python:container:docker` auto run all the python version docker?
>
> What do you think?
>
> [1]
> [https://github.com/apache/beam/commit/47feeafb21023e2a60ae51737cc4000a2033719c#diff-1bc5883bcfcc9e883ab7df09e4dcddb0L63]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[ https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=311710&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311710 ] ASF GitHub Bot logged work on BEAM-7389: Author: ASF GitHub Bot Created on: 12/Sep/19 20:48 Start Date: 12/Sep/19 20:48 Worklog Time Spent: 10m Work Description: davidcavazos commented on issue #9561: [BEAM-7389] Generate notebook for filter URL: https://github.com/apache/beam/pull/9561#issuecomment-531000204 Staged: http://apache-beam-website-pull-requests.storage.googleapis.com/9561/documentation/transforms/python/elementwise/filter/index.html Notebook: https://colab.research.google.com/github/davidcavazos/beam/blob/notebook-filter/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311710) Time Spent: 58h 10m (was: 58h) > Colab examples for element-wise transforms (Python) > --- > > Key: BEAM-7389 > URL: https://issues.apache.org/jira/browse/BEAM-7389 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Rose Nguyen >Assignee: David Cavazos >Priority: Minor > Time Spent: 58h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (BEAM-8220) Use released docker images by default
Kyle Weaver created BEAM-8220: - Summary: Use released docker images by default Key: BEAM-8220 URL: https://issues.apache.org/jira/browse/BEAM-8220 Project: Beam Issue Type: Bug Components: sdk-py-harness Reporter: Kyle Weaver Assignee: Kyle Weaver Now that we have official released docker images [1], we should consider pulling them as the default instead of requiring the user to first build their own docker images, which is more cumbersome and error-prone. Also, all documentation would need to be updated accordingly. [1] [https://hub.docker.com/u/apachebeam] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7389) Colab examples for element-wise transforms (Python)
[ https://issues.apache.org/jira/browse/BEAM-7389?focusedWorklogId=311703&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311703 ] ASF GitHub Bot logged work on BEAM-7389: Author: ASF GitHub Bot Created on: 12/Sep/19 20:43 Start Date: 12/Sep/19 20:43 Worklog Time Spent: 10m Work Description: davidcavazos commented on pull request #9561: [BEAM-7389] Generate notebook for filter URL: https://github.com/apache/beam/pull/9561 First PR introducing the script to generate notebooks from docs using the `Filter` page as an example. R: @aaltay [script/code] R: @rosetn [writing/formatting] Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[ Wordcount 1GB Python PKB benchmarks sometimes fail with uninformative error
[
https://issues.apache.org/jira/browse/BEAM-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928866#comment-16928866
]
Chamikara Jayalath commented on BEAM-8215:
--
Seems like this is making Python 3 post commits extremely flaky (but not Python
2 strangely).
Some recent examples.
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py35/479/console]
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/456/console]
> Wordcount 1GB Python PKB benchmarks sometimes fail with uninformative error
> ---
>
> Key: BEAM-8215
> URL: https://issues.apache.org/jira/browse/BEAM-8215
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Mark Liu
>Priority: Major
>
> Example:
> https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/452/console
> {noformat}
> 12:09:27 2019-09-11 19:09:27,655 a47400ce MainThread
> beam_integration_benchmark(1/1) ERRORError during benchmark
> beam_integration_benchmark
> 12:09:27 Traceback (most recent call last):
> 12:09:27 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
> line 841, in RunBenchmark
> 12:09:27 DoRunPhase(spec, collector, detailed_timer)
> 12:09:27 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
> line 687, in DoRunPhase
> 12:09:27 samples = spec.BenchmarkRun(spec)
> 12:09:27 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
> line 160, in Run
> 12:09:27 job_type=job_type)
> 12:09:27 File
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
> line 91, in SubmitJob
> 12:09:27 assert retcode == 0, "Integration Test Failed."
> 12:09:27 AssertionError: Integration Test Failed.
> {noformat}
> It seems like Job submission failed, but there are no details. I talked with
> [~markflyhigh], and sounds like we plan to stop using PKB in favor of another
> framework.
> Assigning to Mark for now to triage follow up or reassign as appropriate.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (BEAM-8219) crossLanguagePortableWordCount seems to be flaky for beam_PostCommit_Python2
Chamikara Jayalath created BEAM-8219: Summary: crossLanguagePortableWordCount seems to be flaky for beam_PostCommit_Python2 Key: BEAM-8219 URL: https://issues.apache.org/jira/browse/BEAM-8219 Project: Beam Issue Type: Bug Components: test-failures Reporter: Chamikara Jayalath Assignee: Chamikara Jayalath For example, [https://builds.apache.org/job/beam_PostCommit_Python2/451/console] [https://builds.apache.org/job/beam_PostCommit_Python2/454/console] *10:37:22* * What went wrong:*10:37:22* Execution failed for task ':sdks:python:test-suites:portable:py2:crossLanguagePortableWordCount'.*10:37:22* > Process 'command 'sh'' finished with non-zero exit value 1*10:37:22* cc: [~heejong] [~mxm] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8217) py3 sdk harness status server does not print thread stack
[
https://issues.apache.org/jira/browse/BEAM-8217?focusedWorklogId=311665&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311665
]
ASF GitHub Bot logged work on BEAM-8217:
Author: ASF GitHub Bot
Created on: 12/Sep/19 19:53
Start Date: 12/Sep/19 19:53
Worklog Time Spent: 10m
Work Description: angoenka commented on issue #9559: [BEAM-8217]
Cherry-pick PR #9551: Encode str to bytes py3 in sdk harness status server
URL: https://github.com/apache/beam/pull/9559#issuecomment-530980093
Run Portable_Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311665)
Time Spent: 1h (was: 50m)
> py3 sdk harness status server does not print thread stack
> -
>
> Key: BEAM-8217
> URL: https://issues.apache.org/jira/browse/BEAM-8217
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Ankur Goenka
>Priority: Blocker
> Fix For: 2.16.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The calls are failing with
>
> Exception happened during processing of request from ('127.0.0.1', 56106)
> Traceback (most recent call last):
> File "/usr/lib/python3.6/socketserver.py", line 320, in
> _handle_request_noblock
> self.process_request(request, client_address)
> File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
> self.finish_request(request, client_address)
> File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
> self.RequestHandlerClass(request, client_address, self)
> File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
> self.handle()
> File "/usr/lib/python3.6/http/server.py", line 418, in handle
> self.handle_one_request()
> File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
> method()
> File "a.py", line 70, in do_GET
> self.wfile.write(line)
> File "/usr/lib/python3.6/socketserver.py", line 803, in write
> self._sock.sendall(b)
> TypeError: a bytes-like object is required, not 'str'
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (BEAM-8217) py3 sdk harness status server does not print thread stack
[
https://issues.apache.org/jira/browse/BEAM-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928839#comment-16928839
]
Mark Liu commented on BEAM-8217:
https://github.com/apache/beam/pull/9559 is out to cherry-pick this change into
2.16 release branch
> py3 sdk harness status server does not print thread stack
> -
>
> Key: BEAM-8217
> URL: https://issues.apache.org/jira/browse/BEAM-8217
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Ankur Goenka
>Priority: Blocker
> Fix For: 2.16.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The calls are failing with
>
> Exception happened during processing of request from ('127.0.0.1', 56106)
> Traceback (most recent call last):
> File "/usr/lib/python3.6/socketserver.py", line 320, in
> _handle_request_noblock
> self.process_request(request, client_address)
> File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
> self.finish_request(request, client_address)
> File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
> self.RequestHandlerClass(request, client_address, self)
> File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
> self.handle()
> File "/usr/lib/python3.6/http/server.py", line 418, in handle
> self.handle_one_request()
> File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
> method()
> File "a.py", line 70, in do_GET
> self.wfile.write(line)
> File "/usr/lib/python3.6/socketserver.py", line 803, in write
> self._sock.sendall(b)
> TypeError: a bytes-like object is required, not 'str'
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311605&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311605 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 12/Sep/19 19:14 Start Date: 12/Sep/19 19:14 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-530964915 Thanks. I will review the usage of vendor calcite and merge this PR if I don't have further comment. Also FYI vendoring work is done by #9333 @vectorijk would you mind rebasing this PR and pass tests? It would also be great if you could squash this PR a bit into one single commit. I usually do rebase and requash together: ``` git fetch upstream git rebase -i upstream/master ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311605) Time Spent: 8h 40m (was: 8.5h) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 8h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub
[ https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=311607&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311607 ] ASF GitHub Bot logged work on BEAM-5539: Author: ASF GitHub Bot Created on: 12/Sep/19 19:14 Start Date: 12/Sep/19 19:14 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #9558: [BEAM-5539] Cherry-picks PR #9491: Upgrade google-cloud-pubsub and bigtable packages. URL: https://github.com/apache/beam/pull/9558#issuecomment-530966139 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311607) Time Spent: 4h 50m (was: 4h 40m) > Beam Dependency Update Request: google-cloud-pubsub > --- > > Key: BEAM-5539 > URL: https://issues.apache.org/jira/browse/BEAM-5539 > Project: Beam > Issue Type: Bug > Components: dependencies, io-py-gcp >Reporter: Beam JIRA Bot >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 4h 50m > Remaining Estimate: 0h > > - 2018-10-01 19:17:59.633423 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:11:22.339342 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:10:17.795498 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:10:36.369849 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-29 12:12:47.703482 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-05 12:11:03.025934 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-12 12:11:07.165089 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-19 12:11:33.666549 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:09:57.098788 >
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311604&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311604 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 12/Sep/19 19:14 Start Date: 12/Sep/19 19:14 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-530964915 Thanks. I will review the usage of vendor calcite and merge this PR if I don't have further comment. Also FYI vendoring work is done by #9333 @vectorijk would you mind rebasing this PR and pass tests? It would also be great if you could squash this PR a bit into one single commit. I usually do rebase and requash together: ``` git fetch upstream` git rebase -i upstream/master ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311604) Time Spent: 8.5h (was: 8h 20m) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311595&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311595 ] ASF GitHub Bot logged work on BEAM-5820: Author: ASF GitHub Bot Created on: 12/Sep/19 19:11 Start Date: 12/Sep/19 19:11 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #9189: [BEAM-5820] vendor calcite URL: https://github.com/apache/beam/pull/9189#issuecomment-530964915 Thanks. I will review the usage of vendor calcite and merge this PR if I don't have further comment. Also FYI vendoring work is done by #9333 @vectorijk would you mind rebasing this PR and pass tests? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311595) Time Spent: 8h 20m (was: 8h 10m) > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (BEAM-8196) Python 3.5 post commit timed out at 100 minutes
[ https://issues.apache.org/jira/browse/BEAM-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928752#comment-16928752 ] Udi Meiri edited comment on BEAM-8196 at 9/12/19 5:50 PM: -- #9547 was merged ~18 hours ago, but this postcommit timed out about 4 hours ago: https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/console edit: it seems that all (at least 2) of these tests failed due to endpoints_pb2 import failure. was (Author: udim): #9547 was merged ~18 hours ago, but this postcommit timed out about 4 hours ago: https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/console > Python 3.5 post commit timed out at 100 minutes > --- > > Key: BEAM-8196 > URL: https://issues.apache.org/jira/browse/BEAM-8196 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Critical > Time Spent: 40m > Remaining Estimate: 0h > > https://builds.apache.org/job/beam_PostCommit_Python35/435/ > This post commit took 100 minutes and timedout. Should we increase the > timeout? We can also look into why this postcommit was slow. A later post > commit (https://builds.apache.org/job/beam_PostCommit_Python35/437/) > completed in 66 minutes. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub
[ https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=311553&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311553 ] ASF GitHub Bot logged work on BEAM-5539: Author: ASF GitHub Bot Created on: 12/Sep/19 17:47 Start Date: 12/Sep/19 17:47 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #9558: [BEAM-5539] Cherry-picks PR #9491: Upgrade google-cloud-pubsub and bigtable packages. URL: https://github.com/apache/beam/pull/9558#issuecomment-530932872 Portable_Python failure is not related. I'll merge once Python Precommit pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311553) Time Spent: 4h 40m (was: 4.5h) > Beam Dependency Update Request: google-cloud-pubsub > --- > > Key: BEAM-5539 > URL: https://issues.apache.org/jira/browse/BEAM-5539 > Project: Beam > Issue Type: Bug > Components: dependencies, io-py-gcp >Reporter: Beam JIRA Bot >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 4h 40m > Remaining Estimate: 0h > > - 2018-10-01 19:17:59.633423 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:11:22.339342 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:10:17.795498 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:10:36.369849 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-29 12:12:47.703482 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-05 12:11:03.025934 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-12 12:11:07.165089 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-19 12:11:33.666549 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. >
[jira] [Updated] (BEAM-7611) Python BigTableIO IT is not running in any test suites
[ https://issues.apache.org/jira/browse/BEAM-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Jayalath updated BEAM-7611: - Priority: Critical (was: Blocker) > Python BigTableIO IT is not running in any test suites > -- > > Key: BEAM-7611 > URL: https://issues.apache.org/jira/browse/BEAM-7611 > Project: Beam > Issue Type: Bug > Components: io-py-gcp, testing >Reporter: Chamikara Jayalath >Assignee: Solomon Duskis >Priority: Critical > Fix For: 2.16.0 > > > We added an integration test here: [https://github.com/apache/beam/pull/7367] > > But this currently does not get picked up by any test suites (and get skipped > by some due to missing dependencies) hence BigTable sink is largely untested. > > First attempt to enable it: [https://github.com/apache/beam/pull/8886] > > Solomon assigning to you since I cannot find Juan's (PR author) Jira ID. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-7611) Python BigTableIO IT is not running in any test suites
[ https://issues.apache.org/jira/browse/BEAM-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928763#comment-16928763 ] Chamikara Jayalath commented on BEAM-7611: -- Created cherry-pick [https://github.com/apache/beam/pull/9558] Reducing the severity of this bug from a blocker to critical. > Python BigTableIO IT is not running in any test suites > -- > > Key: BEAM-7611 > URL: https://issues.apache.org/jira/browse/BEAM-7611 > Project: Beam > Issue Type: Bug > Components: io-py-gcp, testing >Reporter: Chamikara Jayalath >Assignee: Solomon Duskis >Priority: Blocker > Fix For: 2.16.0 > > > We added an integration test here: [https://github.com/apache/beam/pull/7367] > > But this currently does not get picked up by any test suites (and get skipped > by some due to missing dependencies) hence BigTable sink is largely untested. > > First attempt to enable it: [https://github.com/apache/beam/pull/8886] > > Solomon assigning to you since I cannot find Juan's (PR author) Jira ID. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (BEAM-7611) Python BigTableIO IT is not running in any test suites
[ https://issues.apache.org/jira/browse/BEAM-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Jayalath updated BEAM-7611: - Fix Version/s: (was: 2.16.0) > Python BigTableIO IT is not running in any test suites > -- > > Key: BEAM-7611 > URL: https://issues.apache.org/jira/browse/BEAM-7611 > Project: Beam > Issue Type: Bug > Components: io-py-gcp, testing >Reporter: Chamikara Jayalath >Assignee: Solomon Duskis >Priority: Critical > > We added an integration test here: [https://github.com/apache/beam/pull/7367] > > But this currently does not get picked up by any test suites (and get skipped > by some due to missing dependencies) hence BigTable sink is largely untested. > > First attempt to enable it: [https://github.com/apache/beam/pull/8886] > > Solomon assigning to you since I cannot find Juan's (PR author) Jira ID. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub
[ https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=311537&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311537 ] ASF GitHub Bot logged work on BEAM-5539: Author: ASF GitHub Bot Created on: 12/Sep/19 17:23 Start Date: 12/Sep/19 17:23 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #9558: [BEAM-5539] Cherry-picks PR #9491: Upgrade google-cloud-pubsub and bigtable packages. URL: https://github.com/apache/beam/pull/9558#issuecomment-530923598 R: @markflyhigh CC: @udim This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311537) Time Spent: 4.5h (was: 4h 20m) > Beam Dependency Update Request: google-cloud-pubsub > --- > > Key: BEAM-5539 > URL: https://issues.apache.org/jira/browse/BEAM-5539 > Project: Beam > Issue Type: Bug > Components: dependencies, io-py-gcp >Reporter: Beam JIRA Bot >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 4.5h > Remaining Estimate: 0h > > - 2018-10-01 19:17:59.633423 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:11:22.339342 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:10:17.795498 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:10:36.369849 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-29 12:12:47.703482 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-05 12:11:03.025934 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-12 12:11:07.165089 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-19 12:11:33.666549 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:09:57
[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub
[ https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=311536&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311536 ] ASF GitHub Bot logged work on BEAM-5539: Author: ASF GitHub Bot Created on: 12/Sep/19 17:22 Start Date: 12/Sep/19 17:22 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #9558: [BEAM-5539] Cherry-picks PR #9491: Upgrade google-cloud-pubsub and bigtable packages. URL: https://github.com/apache/beam/pull/9558 Cherry-picks PR #9491: Upgrade google-cloud-pubsub and bigtable packages. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[ Python 3.5 post commit timed out at 100 minutes
[ https://issues.apache.org/jira/browse/BEAM-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928752#comment-16928752 ] Udi Meiri commented on BEAM-8196: - #9547 was merged ~18 hours ago, but this postcommit timed out about 4 hours ago: https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/console > Python 3.5 post commit timed out at 100 minutes > --- > > Key: BEAM-8196 > URL: https://issues.apache.org/jira/browse/BEAM-8196 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Critical > Time Spent: 40m > Remaining Estimate: 0h > > https://builds.apache.org/job/beam_PostCommit_Python35/435/ > This post commit took 100 minutes and timedout. Should we increase the > timeout? We can also look into why this postcommit was slow. A later post > commit (https://builds.apache.org/job/beam_PostCommit_Python35/437/) > completed in 66 minutes. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub
[ https://issues.apache.org/jira/browse/BEAM-5539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-5539. - Resolution: Fixed > Beam Dependency Update Request: google-cloud-pubsub > --- > > Key: BEAM-5539 > URL: https://issues.apache.org/jira/browse/BEAM-5539 > Project: Beam > Issue Type: Bug > Components: dependencies, io-py-gcp >Reporter: Beam JIRA Bot >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 4h 10m > Remaining Estimate: 0h > > - 2018-10-01 19:17:59.633423 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:11:22.339342 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:10:17.795498 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:10:36.369849 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-29 12:12:47.703482 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-05 12:11:03.025934 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-12 12:11:07.165089 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-19 12:11:33.666549 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:09:57.098788 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-03 12:10:44.692853 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.39.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-10 12:10:31.139202 > - >
[jira] [Commented] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub
[ https://issues.apache.org/jira/browse/BEAM-5539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928748#comment-16928748 ] Udi Meiri commented on BEAM-5539: - Should be upgraded to 1.0.0 now. > Beam Dependency Update Request: google-cloud-pubsub > --- > > Key: BEAM-5539 > URL: https://issues.apache.org/jira/browse/BEAM-5539 > Project: Beam > Issue Type: Bug > Components: dependencies, io-py-gcp >Reporter: Beam JIRA Bot >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 4h 10m > Remaining Estimate: 0h > > - 2018-10-01 19:17:59.633423 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:11:22.339342 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:10:17.795498 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.26.0. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:10:36.369849 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-29 12:12:47.703482 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-05 12:11:03.025934 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-12 12:11:07.165089 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-19 12:11:33.666549 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:09:57.098788 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.38.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-03 12:10:44.692853 > - > Please consider upgrading the dependency google-cloud-pubsub. > The current version is 0.35.4. The latest version is 0.39.0 > cc: [~markflyhigh], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > -
[jira] [Work logged] (BEAM-8098) BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK correspond
[ https://issues.apache.org/jira/browse/BEAM-8098?focusedWorklogId=311525&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311525 ] ASF GitHub Bot logged work on BEAM-8098: Author: ASF GitHub Bot Created on: 12/Sep/19 17:04 Start Date: 12/Sep/19 17:04 Worklog Time Spent: 10m Work Description: pabloem commented on issue #9555: [BEAM-8098] Improve documentation on BigQueryIO URL: https://github.com/apache/beam/pull/9555#issuecomment-530916432 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311525) Time Spent: 20m (was: 10m) > BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK > correspond > --- > > Key: BEAM-8098 > URL: https://issues.apache.org/jira/browse/BEAM-8098 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > While working on [https://github.com/apache/beam/pull/9144], I realized there > is a gap in BigQueryIO documentation on mapping between data types defined in > BigQuery and in Beam SDK. > > For example, if a user reads a BYTES field from BigQuery into Beam, it will > be represented as java.nio.ByteBuffer type in Beam Java SDK. The user will > need to do an explicit type cast to ByteBuffer in order to use the data, but > there is no easy way the user can know which type they should cast to, unless > digging into BigQueryIO's implementation (Java - Avro - BigQuery). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual
[ https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=311519&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311519 ] ASF GitHub Bot logged work on BEAM-8105: Author: ASF GitHub Bot Created on: 12/Sep/19 16:52 Start Date: 12/Sep/19 16:52 Worklog Time Spent: 10m Work Description: soyrice commented on issue #9510: [BEAM-8105] update release guide with docker images URL: https://github.com/apache/beam/pull/9510#issuecomment-530912262 > @soyrice I addressed your comments, PTAL. > @markflyhigh , your comments were addressed at the same commit. PTAL. LGTM! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311519) Time Spent: 6h 50m (was: 6h 40m) > Add container publishing instruction to release manual > -- > > Key: BEAM-8105 > URL: https://issues.apache.org/jira/browse/BEAM-8105 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.16.0 > > Time Spent: 6h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8131) Provide Kubernetes setup with Prometheus
[ https://issues.apache.org/jira/browse/BEAM-8131?focusedWorklogId=311514&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311514 ] ASF GitHub Bot logged work on BEAM-8131: Author: ASF GitHub Bot Created on: 12/Sep/19 16:37 Start Date: 12/Sep/19 16:37 Worklog Time Spent: 10m Work Description: aaltay commented on issue #9482: [BEAM-8131] Provide Kubernetes setup for Prometheus URL: https://github.com/apache/beam/pull/9482#issuecomment-530906506 My concerns related to additional infra was addressed. I added @markflyhigh as a reviewer as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311514) Time Spent: 2h 20m (was: 2h 10m) > Provide Kubernetes setup with Prometheus > > > Key: BEAM-8131 > URL: https://issues.apache.org/jira/browse/BEAM-8131 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Kamil Wasilewski >Assignee: Kamil Wasilewski >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8217) py3 sdk harness status server does not print thread stack
[
https://issues.apache.org/jira/browse/BEAM-8217?focusedWorklogId=311499&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311499
]
ASF GitHub Bot logged work on BEAM-8217:
Author: ASF GitHub Bot
Created on: 12/Sep/19 16:22
Start Date: 12/Sep/19 16:22
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9551: [BEAM-8217]
Encode str to bytes py3 in sdk harness status server
URL: https://github.com/apache/beam/pull/9551
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311499)
Time Spent: 40m (was: 0.5h)
> py3 sdk harness status server does not print thread stack
> -
>
> Key: BEAM-8217
> URL: https://issues.apache.org/jira/browse/BEAM-8217
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Ankur Goenka
>Priority: Blocker
> Fix For: 2.16.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The calls are failing with
>
> Exception happened during processing of request from ('127.0.0.1', 56106)
> Traceback (most recent call last):
> File "/usr/lib/python3.6/socketserver.py", line 320, in
> _handle_request_noblock
> self.process_request(request, client_address)
> File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
> self.finish_request(request, client_address)
> File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
> self.RequestHandlerClass(request, client_address, self)
> File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
> self.handle()
> File "/usr/lib/python3.6/http/server.py", line 418, in handle
> self.handle_one_request()
> File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
> method()
> File "a.py", line 70, in do_GET
> self.wfile.write(line)
> File "/usr/lib/python3.6/socketserver.py", line 803, in write
> self._sock.sendall(b)
> TypeError: a bytes-like object is required, not 'str'
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-8217) py3 sdk harness status server does not print thread stack
[
https://issues.apache.org/jira/browse/BEAM-8217?focusedWorklogId=311500&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311500
]
ASF GitHub Bot logged work on BEAM-8217:
Author: ASF GitHub Bot
Created on: 12/Sep/19 16:22
Start Date: 12/Sep/19 16:22
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #9551: [BEAM-8217] Encode str
to bytes py3 in sdk harness status server
URL: https://github.com/apache/beam/pull/9551#issuecomment-530901257
@markflyhigh or @angoenka could one of you cherry pick this to the release
branch?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 311500)
Time Spent: 50m (was: 40m)
> py3 sdk harness status server does not print thread stack
> -
>
> Key: BEAM-8217
> URL: https://issues.apache.org/jira/browse/BEAM-8217
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Ankur Goenka
>Priority: Blocker
> Fix For: 2.16.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The calls are failing with
>
> Exception happened during processing of request from ('127.0.0.1', 56106)
> Traceback (most recent call last):
> File "/usr/lib/python3.6/socketserver.py", line 320, in
> _handle_request_noblock
> self.process_request(request, client_address)
> File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
> self.finish_request(request, client_address)
> File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
> self.RequestHandlerClass(request, client_address, self)
> File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
> self.handle()
> File "/usr/lib/python3.6/http/server.py", line 418, in handle
> self.handle_one_request()
> File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
> method()
> File "a.py", line 70, in do_GET
> self.wfile.write(line)
> File "/usr/lib/python3.6/socketserver.py", line 803, in write
> self._sock.sendall(b)
> TypeError: a bytes-like object is required, not 'str'
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (BEAM-7611) Python BigTableIO IT is not running in any test suites
[ https://issues.apache.org/jira/browse/BEAM-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928695#comment-16928695 ] Ahmet Altay commented on BEAM-7611: --- Yes, it needs to be cherry picked to release branch. > Python BigTableIO IT is not running in any test suites > -- > > Key: BEAM-7611 > URL: https://issues.apache.org/jira/browse/BEAM-7611 > Project: Beam > Issue Type: Bug > Components: io-py-gcp, testing >Reporter: Chamikara Jayalath >Assignee: Solomon Duskis >Priority: Blocker > Fix For: 2.16.0 > > > We added an integration test here: [https://github.com/apache/beam/pull/7367] > > But this currently does not get picked up by any test suites (and get skipped > by some due to missing dependencies) hence BigTable sink is largely untested. > > First attempt to enable it: [https://github.com/apache/beam/pull/8886] > > Solomon assigning to you since I cannot find Juan's (PR author) Jira ID. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928611#comment-16928611 ] Mark Liu edited comment on BEAM-8209 at 9/12/19 3:05 PM: - What work needs to be done here? Can you update on daily base since 2.16 branch was cut and all blockers needs to be resolved before rc vote. was (Author: markflyhigh): What work needs to be done here? Can you update on daily base since 2.16 branch is cut and all blockers needs to be resolved before rc vote. > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928611#comment-16928611 ] Mark Liu edited comment on BEAM-8209 at 9/12/19 3:04 PM: - What work needs to be done here? Can you update on daily base since 2.16 branch is cut and all blockers needs to be resolved before rc vote. was (Author: markflyhigh): What work needs to done here? Can you update on daily base since 2.16 branch is cut and all blockers needs to be resolved before rc vote. > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8209) Document custom docker containers
[ https://issues.apache.org/jira/browse/BEAM-8209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928615#comment-16928615 ] Cyrus Maden commented on BEAM-8209: --- This only requires website changes (shouldn't block the release itself). I'm currently writing a how-to guide about building SDK container images and customizing them with extra Python dependencies, etc. > Document custom docker containers > - > > Key: BEAM-8209 > URL: https://issues.apache.org/jira/browse/BEAM-8209 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Cyrus Maden >Assignee: Cyrus Maden >Priority: Minor > Fix For: 2.16.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-7611) Python BigTableIO IT is not running in any test suites
[ https://issues.apache.org/jira/browse/BEAM-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928614#comment-16928614 ] Mark Liu commented on BEAM-7611: https://github.com/apache/beam/pull/9491 is merged. Does it require cp to release branch? > Python BigTableIO IT is not running in any test suites > -- > > Key: BEAM-7611 > URL: https://issues.apache.org/jira/browse/BEAM-7611 > Project: Beam > Issue Type: Bug > Components: io-py-gcp, testing >Reporter: Chamikara Jayalath >Assignee: Solomon Duskis >Priority: Blocker > Fix For: 2.16.0 > > > We added an integration test here: [https://github.com/apache/beam/pull/7367] > > But this currently does not get picked up by any test suites (and get skipped > by some due to missing dependencies) hence BigTable sink is largely untested. > > First attempt to enable it: [https://github.com/apache/beam/pull/8886] > > Solomon assigning to you since I cannot find Juan's (PR author) Jira ID. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
