[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=358162&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358162
]
ASF GitHub Bot logged work on BEAM-7926:
----------------------------------------
Author: ASF GitHub Bot
Created on: 11/Dec/19 23:07
Start Date: 11/Dec/19 23:07
Worklog Time Spent: 10m
Work Description: KevinGG commented on pull request #10346: [BEAM-7926]
Data-centric Interactive Part2
URL: https://github.com/apache/beam/pull/10346#discussion_r356883895
##########
File path:
sdks/python/apache_beam/runners/interactive/pipeline_fragment_test.py
##########
@@ -0,0 +1,73 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for apache_beam.runners.interactive.pipeline_fragment."""
+from __future__ import absolute_import
+
+import unittest
+
+import apache_beam as beam
+from apache_beam.runners.interactive import interactive_beam as ib
+from apache_beam.runners.interactive import interactive_environment as ie
+from apache_beam.runners.interactive import interactive_runner as ir
+from apache_beam.runners.interactive import pipeline_fragment as pf
+from apache_beam.runners.interactive.testing.mock_ipython import
mock_get_ipython
+from apache_beam.runners.interactive.testing.pipeline_assertion import
assert_pipeline_equal
+
+# TODO(BEAM-8288): clean up the work-around of nose tests using Python2 without
+# unittest.mock module.
+try:
+ from unittest.mock import patch
+except ImportError:
+ from mock import patch
+
+
[email protected](not ie.current_env().is_interactive_ready,
+ '[interactive] dependency is not installed.')
+class PipelineFragmentTest(unittest.TestCase):
+
+ def setUp(self):
+ # Assume a notebook frontend is connected to the mocked ipython kernel.
+ ie.current_env()._is_in_ipython = True
+ ie.current_env()._is_in_notebook = True
+
+ @patch('IPython.get_ipython', mock_get_ipython)
+ def test_build_pipeline_fragment(self):
+ with mock_get_ipython(): # Cell 1
+ p = beam.Pipeline(ir.InteractiveRunner())
+ p_expected = beam.Pipeline(ir.InteractiveRunner())
+ # Watch local scope now to allow interactive beam to track the pipelines.
+ ib.watch(locals())
+
+ with mock_get_ipython(): # Cell 2
+ # pylint: disable=range-builtin-not-iterating
+ init = p | 'Init' >> beam.Create(range(10))
+ init_expected = p_expected | 'Init' >> beam.Create(range(10))
+
+ with mock_get_ipython(): # Cell 3
+ square = init | 'Square' >> beam.Map(lambda x: x * x)
+ _ = init | 'Cube' >> beam.Map(lambda x: x ** 3)
+ _ = init_expected | 'Square' >> beam.Map(lambda x: x * x)
+
+ # Watch every PCollections have been defined so far in local scope.
+ ib.watch(locals())
+ fragment = pf.PipelineFragment([square]).deduce_fragment()
+ assert_pipeline_equal(self, p_expected, fragment)
+
Review comment:
Added the test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 358162)
Time Spent: 29h 40m (was: 29.5h)
> Show PCollection with Interactive Beam in a data-centric user flow
> ------------------------------------------------------------------
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
> Reporter: Ning Kang
> Assignee: Ning Kang
> Priority: Major
> Time Spent: 29h 40m
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
