[jira] [Created] (BEAM-678) Python executable should be overridable

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-678:
-

 Summary: Python executable should be overridable
 Key: BEAM-678
 URL: https://issues.apache.org/jira/browse/BEAM-678
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Scott Wegner
Assignee: Scott Wegner
Priority: Minor


Since [PR 962|https://github.com/apache/incubator-beam/pull/962], dependency.py 
will use {{sys.executable}} to source the python executable to use for managing 
dependencies. However, as stated in the [python 
docs|https://docs.python.org/2/library/sys.html#sys.executable]:

bq. If Python is unable to retrieve the real path to its executable, 
sys.executable will be an empty string or None.

To mitigate this, we should allow setting the python executable to use via an 
environment variable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1000: Avro sinks

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/incubator-beam/pull/1000


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1212

[Apache Jenkins Server]

See 

[2/2] incubator-beam git commit: Closes #1006

[robertwb]

Closes #1006


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bb649061
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bb649061
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bb649061

Branch: refs/heads/python-sdk
Commit: bb649061f46867c9721e82963e91ac6fe1387245
Parents: 753cc9c 26eb336
Author: Robert Bradshaw 
Authored: Mon Sep 26 10:58:15 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Sep 26 10:58:15 2016 -0700

--
 sdks/python/apache_beam/utils/dependency.py | 16 +---
 1 file changed, 13 insertions(+), 3 deletions(-)
--

[2/4] incubator-beam git commit: Fix and add test for ReadFromAvro transform.

[robertwb]

Fix and add test for ReadFromAvro transform.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8bc965b6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8bc965b6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8bc965b6

Branch: refs/heads/python-sdk
Commit: 8bc965b61becf5f92bcd3ff4468fa53fde5b6e6b
Parents: f9c565b
Author: Robert Bradshaw 
Authored: Sat Sep 24 01:26:31 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Sep 26 12:17:35 2016 -0700

--
 sdks/python/apache_beam/io/avroio.py  | 5 ++---
 sdks/python/apache_beam/io/avroio_test.py | 8 
 2 files changed, 10 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8bc965b6/sdks/python/apache_beam/io/avroio.py
--
diff --git a/sdks/python/apache_beam/io/avroio.py 
b/sdks/python/apache_beam/io/avroio.py
index 3415d22..82b30be 100644
--- a/sdks/python/apache_beam/io/avroio.py
+++ b/sdks/python/apache_beam/io/avroio.py
@@ -34,8 +34,7 @@ __all__ = ['ReadFromAvro']
 class ReadFromAvro(PTransform):
   """A ``PTransform`` for reading avro files."""
 
-  def __init__(self, label=None, file_pattern=None, min_bundle_size=0,
-   **kwargs):
+  def __init__(self, file_pattern=None, min_bundle_size=0):
 """Initializes ``ReadFromAvro``.
 
 Uses source '_AvroSource' to read a set of Avro files defined by a given
@@ -70,7 +69,7 @@ class ReadFromAvro(PTransform):
splitting the input into bundles.
   **kwargs: Additional keyword arguments to be passed to the base class.
 """
-super(ReadFromAvro, self).__init__(label, **kwargs)
+super(ReadFromAvro, self).__init__()
 
 self._file_pattern = file_pattern
 self._min_bundle_size = min_bundle_size

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8bc965b6/sdks/python/apache_beam/io/avroio_test.py
--
diff --git a/sdks/python/apache_beam/io/avroio_test.py 
b/sdks/python/apache_beam/io/avroio_test.py
index c21ed57..e0c211f 100644
--- a/sdks/python/apache_beam/io/avroio_test.py
+++ b/sdks/python/apache_beam/io/avroio_test.py
@@ -20,8 +20,12 @@ import os
 import tempfile
 import unittest
 
+import apache_beam as beam
+from apache_beam.io import avroio
 from apache_beam.io import filebasedsource
 from apache_beam.io import source_test_utils
+from apache_beam.transforms.util import assert_that
+from apache_beam.transforms.util import equal_to
 
 # Importing following private class for testing purposes.
 from apache_beam.io.avroio import _AvroSource as AvroSource
@@ -223,6 +227,10 @@ class TestAvro(unittest.TestCase):
   source_test_utils.readFromSource(source, None, None)
   self.assertEqual(0, exn.exception.message.find('Unexpected sync marker'))
 
+  def test_pipeline(self):
+path = self._write_data()
+with beam.Pipeline('DirectPipelineRunner') as p:
+  assert_that(p | avroio.ReadFromAvro(path), equal_to(self.RECORDS))
 
 if __name__ == '__main__':
   logging.getLogger().setLevel(logging.INFO)

[1/4] incubator-beam git commit: Implement avro sink.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 7d988e3bb -> 7e744e445


Implement avro sink.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1090ca39
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1090ca39
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1090ca39

Branch: refs/heads/python-sdk
Commit: 1090ca3911faef84bbd61c6b41668a695d2fe8dc
Parents: 8bc965b
Author: Robert Bradshaw 
Authored: Sat Sep 24 02:25:05 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Sep 26 12:17:35 2016 -0700

--
 sdks/python/apache_beam/io/avroio.py  | 100 -
 sdks/python/apache_beam/io/avroio_test.py |  41 ++
 2 files changed, 127 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1090ca39/sdks/python/apache_beam/io/avroio.py
--
diff --git a/sdks/python/apache_beam/io/avroio.py 
b/sdks/python/apache_beam/io/avroio.py
index 82b30be..fdf8dae 100644
--- a/sdks/python/apache_beam/io/avroio.py
+++ b/sdks/python/apache_beam/io/avroio.py
@@ -20,15 +20,18 @@ import os
 import StringIO
 import zlib
 
+import avro
 from avro import datafile
 from avro import io as avroio
 from avro import schema
 
+import apache_beam as beam
 from apache_beam.io import filebasedsource
+from apache_beam.io import fileio
 from apache_beam.io.iobase import Read
 from apache_beam.transforms import PTransform
 
-__all__ = ['ReadFromAvro']
+__all__ = ['ReadFromAvro', 'WriteToAvro']
 
 
 class ReadFromAvro(PTransform):
@@ -242,3 +245,98 @@ class _AvroSource(filebasedsource.FileBasedSource):
 sync_marker)
 for record in block.records():
   yield record
+
+
+_avro_codecs = {
+fileio.CompressionTypes.UNCOMPRESSED: 'null',
+fileio.CompressionTypes.ZLIB: 'deflate',
+# fileio.CompressionTypes.SNAPPY: 'snappy',
+}
+
+
+class WriteToAvro(beam.transforms.PTransform):
+  """A ``PTransform`` for writing avro files."""
+
+  def __init__(self,
+   file_path_prefix,
+   schema,
+   file_name_suffix='',
+   num_shards=0,
+   shard_name_template=None,
+   mime_type='application/x-avro',
+   compression_type=fileio.CompressionTypes.ZLIB):
+"""Initialize a WriteToAvro transform.
+
+Args:
+  file_path_prefix: The file path to write to. The files written will begin
+with this prefix, followed by a shard identifier (see num_shards), and
+end in a common extension, if given by file_name_suffix. In most cases,
+only this argument is specified and num_shards, shard_name_template, 
and
+file_name_suffix use default values.
+  schema: The schema to use, as returned by avro.schema.parse
+  file_name_suffix: Suffix for the files written.
+  append_trailing_newlines: indicate whether this sink should write an
+additional newline char after writing each element.
+  num_shards: The number of files (shards) used for output. If not set, the
+service will decide on the optimal number of shards.
+Constraining the number of shards is likely to reduce
+the performance of a pipeline.  Setting this value is not recommended
+unless you require a specific number of output files.
+  shard_name_template: A template string containing placeholders for
+the shard number and shard count. Currently only '' and
+'-S-of-N' are patterns accepted by the service.
+When constructing a filename for a particular shard number, the
+upper-case letters 'S' and 'N' are replaced with the 0-padded shard
+number and shard count respectively.  This argument can be '' in which
+case it behaves as if num_shards was set to 1 and only one file will be
+generated. The default pattern used is '-S-of-N'.
+  mime_type: The MIME type to use for the produced files, if the filesystem
+supports specifying MIME types.
+  compression_type: Used to handle compressed output files. Defaults to
+CompressionTypes.ZLIB
+
+Returns:
+  A WriteToAvro transform usable for writing.
+"""
+if compression_type not in _avro_codecs:
+  raise ValueError(
+  'Compression type %s not supported by avro.' % compression_type)
+self.args = (file_path_prefix, schema, file_name_suffix, num_shards,
+ shard_name_template, mime_type, compression_type)
+
+  def apply(self, pcoll):
+# pylint: disable=expression-not-assigned
+pcoll | beam.io.iobase.Write(_AvroSink(*self.args))
+
+
+class 

[4/4] incubator-beam git commit: Closes #1000

[robertwb]

Closes #1000


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7e744e44
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7e744e44
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7e744e44

Branch: refs/heads/python-sdk
Commit: 7e744e4455cecb08870145ae100d2b8de8bd7134
Parents: 7d988e3 1090ca3
Author: Robert Bradshaw 
Authored: Mon Sep 26 12:17:36 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Sep 26 12:17:36 2016 -0700

--
 sdks/python/apache_beam/io/avroio.py  | 105 -
 sdks/python/apache_beam/io/avroio_test.py |  56 ++---
 sdks/python/apache_beam/io/textio_test.py |   5 ++
 3 files changed, 150 insertions(+), 16 deletions(-)
--

[3/4] incubator-beam git commit: Cleanup temporary files in textio and avroio tests.

[robertwb]

Cleanup temporary files in textio and avroio tests.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f9c565b6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f9c565b6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f9c565b6

Branch: refs/heads/python-sdk
Commit: f9c565b6d9c2b7b11e6cba8f1f7354fd4e369193
Parents: 7d988e3
Author: Robert Bradshaw 
Authored: Sat Sep 24 01:16:43 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Sep 26 12:17:35 2016 -0700

--
 sdks/python/apache_beam/io/avroio_test.py | 9 +
 sdks/python/apache_beam/io/textio_test.py | 5 +
 2 files changed, 14 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f9c565b6/sdks/python/apache_beam/io/avroio_test.py
--
diff --git a/sdks/python/apache_beam/io/avroio_test.py 
b/sdks/python/apache_beam/io/avroio_test.py
index 29c4209..c21ed57 100644
--- a/sdks/python/apache_beam/io/avroio_test.py
+++ b/sdks/python/apache_beam/io/avroio_test.py
@@ -34,11 +34,19 @@ import avro.schema as avro_schema
 
 class TestAvro(unittest.TestCase):
 
+  _temp_files = []
+
   def setUp(self):
 # Reducing the size of thread pools. Without this test execution may fail 
in
 # environments with limited amount of resources.
 filebasedsource.MAX_NUM_THREADS_FOR_SIZE_ESTIMATION = 2
 
+  def tearDown(self):
+for path in self._temp_files:
+  if os.path.exists(path):
+os.remove(path)
+self._temp_files = []
+
   RECORDS = [{'name': 'Thomas',
   'favorite_number': 1,
   'favorite_color': 'blue'}, {'name': 'Henry',
@@ -79,6 +87,7 @@ class TestAvro(unittest.TestCase):
 writer.append(self.RECORDS[i % len_records])
   writer.close()
 
+  self._temp_files.append(f.name)
   return f.name
 
   def _write_pattern(self, num_files):

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f9c565b6/sdks/python/apache_beam/io/textio_test.py
--
diff --git a/sdks/python/apache_beam/io/textio_test.py 
b/sdks/python/apache_beam/io/textio_test.py
index 3fa0f9a..fef2b79 100644
--- a/sdks/python/apache_beam/io/textio_test.py
+++ b/sdks/python/apache_beam/io/textio_test.py
@@ -20,6 +20,7 @@
 import glob
 import gzip
 import logging
+import os
 import tempfile
 import unittest
 import zlib
@@ -330,6 +331,10 @@ class TextSinkTest(unittest.TestCase):
 self.lines = ['Line %d' % d for d in range(100)]
 self.path = tempfile.NamedTemporaryFile().name
 
+  def tearDown(self):
+if os.path.exists(self.path):
+  os.remove(self.path)
+
   def _write_lines(self, sink, lines):
 f = sink.open(self.path)
 for line in lines:

[GitHub] incubator-beam pull request #1007: Add support for bz2 compression in python

[tim1357]

GitHub user tim1357 opened a pull request:

https://github.com/apache/incubator-beam/pull/1007

Add support for bz2 compression in python

This is my first contribution to any open source project ever, so I have no 
idea if I'm doing this right. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tim1357/incubator-beam python-sdk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1007.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1007


commit 5da9ee7e0528e252ee0dd1982228f42c1ca9672d
Author: tim1357 
Date:   2016-09-26T19:15:51Z

Add support for bz2 compression




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Add BEAM_PYTHON environment override to set the python executable

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 753cc9c2e -> bb649061f


Add BEAM_PYTHON environment override to set the python executable


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/26eb336a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/26eb336a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/26eb336a

Branch: refs/heads/python-sdk
Commit: 26eb336ae63dceefae876d6bc8ff742d2670567b
Parents: 753cc9c
Author: Scott Wegner 
Authored: Mon Sep 26 10:05:01 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Sep 26 10:58:11 2016 -0700

--
 sdks/python/apache_beam/utils/dependency.py | 16 +---
 1 file changed, 13 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/26eb336a/sdks/python/apache_beam/utils/dependency.py
--
diff --git a/sdks/python/apache_beam/utils/dependency.py 
b/sdks/python/apache_beam/utils/dependency.py
index e9d73ae..150b2fc 100644
--- a/sdks/python/apache_beam/utils/dependency.py
+++ b/sdks/python/apache_beam/utils/dependency.py
@@ -189,13 +189,23 @@ def _stage_extra_packages(extra_packages, 
staging_location, temp_dir,
   return resources
 
 
+def _get_python_executable():
+  # Allow overriding the python executable to use for downloading and
+  # installing dependencies, otherwise use the python executable for
+  # the current process.
+  beam_python = os.environ.get('BEAM_PYTHON') or sys.executable
+  if not python_bin:
+raise ValueError('Could not find Python executable.')
+  return python_bin
+
+
 def _populate_requirements_cache(requirements_file, cache_dir):
   # The 'pip download' command will not download again if it finds the
   # tarball with the proper version already present.
   # It will get the packages downloaded in the order they are presented in
   # the requirements file and will not download package dependencies.
   cmd_args = [
-  sys.executable, '-m', 'pip', 'install', '--download', cache_dir,
+  _get_python_executable(), '-m', 'pip', 'install', '--download', 
cache_dir,
   '-r', requirements_file,
   # Download from PyPI source distributions.
   '--no-binary', ':all:']
@@ -375,7 +385,7 @@ def _build_setup_package(setup_file, temp_dir, 
build_setup_args=None):
 os.chdir(os.path.dirname(setup_file))
 if build_setup_args is None:
   build_setup_args = [
-  sys.executable, os.path.basename(setup_file),
+  _get_python_executable(), os.path.basename(setup_file),
   'sdist', '--dist-dir', temp_dir]
 logging.info('Executing command: %s', build_setup_args)
 processes.check_call(build_setup_args)
@@ -461,7 +471,7 @@ def _download_pypi_sdk_package(temp_dir):
   version = pkg.get_distribution(GOOGLE_PACKAGE_NAME).version
   # Get a source distribution for the SDK package from PyPI.
   cmd_args = [
-  sys.executable, '-m', 'pip', 'install', '--download', temp_dir,
+  _get_python_executable(), '-m', 'pip', 'install', '--download', temp_dir,
   '%s==%s' % (GOOGLE_PACKAGE_NAME, version),
   '--no-binary', ':all:', '--no-deps']
   logging.info('Executing command: %s', cmd_args)

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1397

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1397

[Apache Jenkins Server]

See 

[2/2] incubator-beam git commit: Closes #1004

[robertwb]

Closes #1004


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7d988e3b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7d988e3b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7d988e3b

Branch: refs/heads/python-sdk
Commit: 7d988e3bb0eab67d1519f5c3a5d0a8b54cb3b88d
Parents: bb64906 7542904
Author: Robert Bradshaw 
Authored: Mon Sep 26 12:16:21 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Sep 26 12:16:21 2016 -0700

--
 sdks/python/.gitignore | 1 +
 1 file changed, 1 insertion(+)
--

[1/2] incubator-beam git commit: Ignore virtualenv environment in git

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk bb649061f -> 7d988e3bb


Ignore virtualenv environment in git


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7542904a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7542904a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7542904a

Branch: refs/heads/python-sdk
Commit: 7542904a1ca05d973cbb32f1df5d739fef29d916
Parents: bb64906
Author: Scott Wegner 
Authored: Mon Sep 26 08:59:18 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Sep 26 12:16:20 2016 -0700

--
 sdks/python/.gitignore | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7542904a/sdks/python/.gitignore
--
diff --git a/sdks/python/.gitignore b/sdks/python/.gitignore
index c03b617..05ed303 100644
--- a/sdks/python/.gitignore
+++ b/sdks/python/.gitignore
@@ -5,6 +5,7 @@
 build/
 dist/
 distribute-*
+env/
 
 # Test files
 .tox/

[jira] [Commented] (BEAM-517) Check versions of pip and cython

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523923#comment-15523923
 ] 

Ahmet Altay commented on BEAM-517:
--

Adding that snippet makes sense. 

[~robertwb] (cython and Beam developer) could comment on the oldest required 
version of cython. 

> Check versions of pip and cython
> 
>
> Key: BEAM-517
> URL: https://issues.apache.org/jira/browse/BEAM-517
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> Python SDK depends on pip and cython however it does not check the versions 
> of these.
> Some of the pip flags does not exist in older versions:
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/28#issuecomment-236382953
> (Note: Even though the above issue was reported by the user in a different 
> repo it is related to the apache beam sdk)
> Similarly with cython, SDK supports running with or without Cython. Because 
> of that reason it is not list it as a requirement in the setup.py file. 
> However, with an old version of cython SDK might fail.
> To avoid the above problem: In the SDK check the version of these packages 
> and show a warning to upgrade.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-615) Add Support for Processing-Time Timers in FlinkRunner Window Operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523377#comment-15523377
 ] 

ASF GitHub Bot commented on BEAM-615:
-

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/994


> Add Support for Processing-Time Timers in FlinkRunner Window Operator
> -
>
> Key: BEAM-615
> URL: https://issues.apache.org/jira/browse/BEAM-615
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating, 0.3.0-incubating
>Reporter: Aljoscha Krettek
>Assignee: Kenneth Knowles
>
> The Flink runner never had support for processing-time timers, they are 
> silently ignored when a trigger tries to set one. This should be easy to add 
> in {{WindowFnOperator}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-674) Add GridFS support to MongoDB IO

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp updated BEAM-674:
-
Component/s: sdk-java-extensions

> Add GridFS support to MongoDB IO
> 
>
> Key: BEAM-674
> URL: https://issues.apache.org/jira/browse/BEAM-674
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Daniel Kulp
>Assignee: Daniel Kulp
>
> MongoDB has an "extension" called GridFS that allows storing of very large 
> "files" into the MongoDB database in a relatively efficient way.   It would 
> be good to add a GridFS API based IO to allow retrieving the data for 
> processing. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-675) Introduce message mapper in JmsIO

[JIRA]

Jean-Baptiste Onofré created BEAM-675:
-

 Summary: Introduce message mapper in JmsIO
 Key: BEAM-675
 URL: https://issues.apache.org/jira/browse/BEAM-675
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


Currently, the {{JmsIO}} deals with a {{JmsRecord}} and populates the reader 
populates the {{PCollection}} with it.

It also deals "only" with {{TextMessage}}.

It would be helpful to allow users to provide {{MessageMapper}} function 
allowing to "convert" the JMS Message as a custom POJO, and eventually deal 
with other type of JMS Message ({{BytesMessage}}, {{MapMessage}}, ...).

By default, the IO will still use {{JmsRecord}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-674) Add GridFS support to MongoDB IO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-674:
--
Assignee: Daniel Kulp  (was: Jean-Baptiste Onofré)

> Add GridFS support to MongoDB IO
> 
>
> Key: BEAM-674
> URL: https://issues.apache.org/jira/browse/BEAM-674
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Daniel Kulp
>Assignee: Daniel Kulp
>
> MongoDB has an "extension" called GridFS that allows storing of very large 
> "files" into the MongoDB database in a relatively efficient way.   It would 
> be good to add a GridFS API based IO to allow retrieving the data for 
> processing. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-674) Add GridFS support to MongoDB IO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523309#comment-15523309
 ] 

Jean-Baptiste Onofré commented on BEAM-674:
---

Ready to review.

> Add GridFS support to MongoDB IO
> 
>
> Key: BEAM-674
> URL: https://issues.apache.org/jira/browse/BEAM-674
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Daniel Kulp
>Assignee: Jean-Baptiste Onofré
>
> MongoDB has an "extension" called GridFS that allows storing of very large 
> "files" into the MongoDB database in a relatively efficient way.   It would 
> be good to add a GridFS API based IO to allow retrieving the data for 
> processing. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-674) Add GridFS support to MongoDB IO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523335#comment-15523335
 ] 

ASF GitHub Bot commented on BEAM-674:
-

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/1003

[BEAM-674] Source part of GridFS IO

This is the "Source" part for GridFS based IO for beam.  (will work on Sink 
next, but would like to get this reviewed and merged first) . The "default" is 
to parse each file as text files (by line), but a parser function can be 
provided to take the InputStream and parse via whatever is required.   

For runners that can split into bundles it attempts to assign files in the 
grid to different bundles.  


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam gridfs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1003.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1003


commit d5cdc2429622f65a762774de8b5baf15334e55e2
Author: Daniel Kulp 
Date:   2016-09-16T20:58:56Z

Add GridFS io

commit a9212662744c14f10cd811540c3e9268c32c25c4
Author: Daniel Kulp 
Date:   2016-09-16T21:19:50Z

Fix checkstyle issues

commit cee0a06b6a465a276c2c5410d7d3f9af703982d4
Author: Daniel Kulp 
Date:   2016-09-19T17:22:44Z

Attempt to get a converter in there

commit fafa8fa607f22eacf918abb13419f28df9d2a8e9
Author: Daniel Kulp 
Date:   2016-09-19T17:32:44Z

Fix javac compile problem

commit 7e9872f12c74902f1a23e5a27eb0027ae753947a
Author: Daniel Kulp 
Date:   2016-09-19T17:50:11Z

Force a serializable

commit 265747946864b226235ee5b758e6c10b7cc3992f
Author: Daniel Kulp 
Date:   2016-09-19T17:56:03Z

Add the needed coder

commit 4f54495afe7ff4768d873350c345d39905d812fc
Author: Daniel Kulp 
Date:   2016-09-19T18:02:39Z

Change to using the GridFSDBFile instead of InputStream so the parsingFn 
can have access to tall the metadata

commit cbeebf02542a5e5a5f4b9a6c370b1b68b46d2deb
Author: Daniel Kulp 
Date:   2016-09-19T18:25:23Z

Flip to allowing the parser to have complete control over how the item is 
added to the collection

commit a08007b9f444fedcde78ab38c6cdf505b3864c61
Author: Daniel Kulp 
Date:   2016-09-19T18:26:19Z

Fix unused imports

commit a4840e98d891d3fa783654a472af06c4d399a929
Author: Daniel Kulp 
Date:   2016-09-21T19:51:45Z

Add test for the parser functionality and cleanup some of that code

commit 438a792a796be77186d79aa3fdb221efcced6d4f
Author: Daniel Kulp 
Date:   2016-09-21T20:01:33Z

Move the coder out from the parser

commit e8fcdbf3cebd6fa4648f328484dee07fec35b21a
Author: Daniel Kulp 
Date:   2016-09-22T12:36:50Z

Fix test

commit 1d1a373fc7cec4e78bf0e618a902c15005fc36b4
Author: Daniel Kulp 
Date:   2016-09-23T14:49:53Z

Flip to using BoundedSource so it can be broken up into bundles




> Add GridFS support to MongoDB IO
> 
>
> Key: BEAM-674
> URL: https://issues.apache.org/jira/browse/BEAM-674
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Daniel Kulp
>Assignee: Daniel Kulp
>
> MongoDB has an "extension" called GridFS that allows storing of very large 
> "files" into the MongoDB database in a relatively efficient way.   It would 
> be good to add a GridFS API based IO to allow retrieving the data for 
> processing. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #994: [BEAM-615] Add support for processing time...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/994


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-677) Add element modification check to Python Beam

[Robert Bradshaw (JIRA)]

Robert Bradshaw created BEAM-677:


 Summary: Add element modification check to Python Beam
 Key: BEAM-677
 URL: https://issues.apache.org/jira/browse/BEAM-677
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Robert Bradshaw
Assignee: Frances Perry


This can be done similarly to java, but encoding the element before and 
comparing the result. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-674) Add GridFS support to MongoDB IO

[Daniel Kulp (JIRA)]

Daniel Kulp created BEAM-674:


 Summary: Add GridFS support to MongoDB IO
 Key: BEAM-674
 URL: https://issues.apache.org/jira/browse/BEAM-674
 Project: Beam
  Issue Type: Bug
Reporter: Daniel Kulp
Assignee: Daniel Kulp



MongoDB has an "extension" called GridFS that allows storing of very large 
"files" into the MongoDB database in a relatively efficient way.   It would be 
good to add a GridFS API based IO to allow retrieving the data for processing. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1005: Clear requirements cache directory before...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1005

Clear requirements cache directory before using it in test

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

`test_with_requirements_file` was failing locally because I had stale files 
in my /tmp/dataflow-requirements-cache from a previous package install. This 
change cleans the cache directory before validating its contents.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam 
clear-requirements-cache

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1005.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1005


commit ea1d367aacd69ae49d9ac486cbd3c07c269017af
Author: Scott Wegner 
Date:   2016-09-26T16:42:47Z

Clear requirements cache directory before using it in test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-614) File compression/decompression should support auto detection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523288#comment-15523288
 ] 

ASF GitHub Bot commented on BEAM-614:
-

GitHub user chamikaramj opened a pull request:

https://github.com/apache/incubator-beam/pull/1002

[BEAM-614] Updates FileBasedSource to support CompressionType.AUTO.

Updates FileBasedSource to support CompressionType.AUTO.

Fixes some tests that were not properly being tested.

Adds tests for CompressionType.AUTO.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam 
fix_compressed_auto_2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1002.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1002


commit 37aa7a457be0c8bfef0bb5f1b6cc973ead93f7b1
Author: chamik...@google.com 
Date:   2016-09-26T04:44:34Z

Updates filebasedsource to support CompressionType.AUTO.




> File compression/decompression should support auto detection
> 
>
> Key: BEAM-614
> URL: https://issues.apache.org/jira/browse/BEAM-614
> Project: Beam
>  Issue Type: Bug
>Reporter: Slaven Bilac
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1002: [BEAM-614] Updates FileBasedSource to sup...

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/incubator-beam/pull/1002

[BEAM-614] Updates FileBasedSource to support CompressionType.AUTO.

Updates FileBasedSource to support CompressionType.AUTO.

Fixes some tests that were not properly being tested.

Adds tests for CompressionType.AUTO.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam 
fix_compressed_auto_2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1002.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1002


commit 37aa7a457be0c8bfef0bb5f1b6cc973ead93f7b1
Author: chamik...@google.com 
Date:   2016-09-26T04:44:34Z

Updates filebasedsource to support CompressionType.AUTO.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-674) Add GridFS support to MongoDB IO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-674:
-

Assignee: Jean-Baptiste Onofré  (was: Daniel Kulp)

> Add GridFS support to MongoDB IO
> 
>
> Key: BEAM-674
> URL: https://issues.apache.org/jira/browse/BEAM-674
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Daniel Kulp
>Assignee: Jean-Baptiste Onofré
>
> MongoDB has an "extension" called GridFS that allows storing of very large 
> "files" into the MongoDB database in a relatively efficient way.   It would 
> be good to add a GridFS API based IO to allow retrieving the data for 
> processing. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-678) Python executable should be overridable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523625#comment-15523625
 ] 

ASF GitHub Bot commented on BEAM-678:
-

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1006

[BEAM-678] Add BEAM_PYTHON environment override to set the python executable

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam set-python-executable

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1006.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1006


commit 046e9b0139ee989d1821036eddb7d0b93a4ae246
Author: Scott Wegner 
Date:   2016-09-26T17:05:01Z

Add BEAM_PYTHON environment override to set the python executable




> Python executable should be overridable
> ---
>
> Key: BEAM-678
> URL: https://issues.apache.org/jira/browse/BEAM-678
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> Since [PR 962|https://github.com/apache/incubator-beam/pull/962], 
> dependency.py will use {{sys.executable}} to source the python executable to 
> use for managing dependencies. However, as stated in the [python 
> docs|https://docs.python.org/2/library/sys.html#sys.executable]:
> bq. If Python is unable to retrieve the real path to its executable, 
> sys.executable will be an empty string or None.
> To mitigate this, we should allow setting the python executable to use via an 
> environment variable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1007: Add support for bz2 compression in python

[tim1357]

GitHub user tim1357 reopened a pull request:

https://github.com/apache/incubator-beam/pull/1007

Add support for bz2 compression in python

This is my first contribution to any open source project ever, so I have no 
idea if I'm doing this right. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tim1357/incubator-beam python-sdk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1007.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1007


commit 5da9ee7e0528e252ee0dd1982228f42c1ca9672d
Author: tim1357 
Date:   2016-09-26T19:15:51Z

Add support for bz2 compression

commit b83e413e6ece1a1178fa566f37fba5164ff96899
Author: tim1357 
Date:   2016-09-26T19:36:50Z

fix style

commit 5b170aa2b02c6ac033beb71c96b18a75a24cc553
Author: tim1357 
Date:   2016-09-26T19:59:40Z

this style checker :/ :(




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-679) Bigtable IO integration tests are failing

[JIRA]

Jean-Baptiste Onofré created BEAM-679:
-

 Summary: Bigtable IO integration tests are failing
 Key: BEAM-679
 URL: https://issues.apache.org/jira/browse/BEAM-679
 Project: Beam
  Issue Type: Bug
  Components: build-system, sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré
Priority: Critical


Bigtable ITests are failing with the following issue:

{code}
java.lang.NoClassDefFoundError: Could not initialize class 
com.google.cloud.bigtable.grpc.BigtableSessionSharedThreadPools 
{code}

I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1008: Fix python bin test.

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/incubator-beam/pull/1008

Fix python bin test.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam python-bin-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1008.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1008


commit 4b4ad7328b5c9528f2aca2f2b8627ae0793cf1b6
Author: Robert Bradshaw 
Date:   2016-09-26T21:14:03Z

Fix python bin test.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #853: [BEAM-409] Ensure precision in division of...

[JasonMWhite]

Github user JasonMWhite closed the pull request at:

https://github.com/apache/incubator-beam/pull/853


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-409) Incorrect use of Math.ceil in ApproximateQuantiles

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524220#comment-15524220
 ] 

ASF GitHub Bot commented on BEAM-409:
-

Github user JasonMWhite closed the pull request at:

https://github.com/apache/incubator-beam/pull/853


> Incorrect use of Math.ceil in ApproximateQuantiles
> --
>
> Key: BEAM-409
> URL: https://issues.apache.org/jira/browse/BEAM-409
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Minor
>  Labels: findbugs, newbie, starter
>
> [FindBugs 
> ICAST_INT_CAST_TO_DOUBLE_PASSED_TO_CEIL|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml#L274]:
>  Integral value cast to double and then passed to Math.ceil
> Applies to: 
> [ApproximateQuanitles|https://github.com/apache/incubator-beam/blob/58a029a06aea1030279e5da8f9fa3114f456c1db/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java#L349].
> This is a good starter bug. When fixing, please remove the corresponding 
> entries from 
> [findbugs-filter.xml|https://github.com/apache/incubator-beam/blob/master/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml]
>  and verify the build passes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-517) Check versions of pip and cython

[Robert Bradshaw (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523965#comment-15523965
 ] 

Robert Bradshaw commented on BEAM-517:
--

+1, please file a PR

I don't think we rely on any recent version of Cython, maybe just ask for 
0.23.2 (released just over a year ago). Might also be worth using 
distutils.version for comparisons. 

> Check versions of pip and cython
> 
>
> Key: BEAM-517
> URL: https://issues.apache.org/jira/browse/BEAM-517
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> Python SDK depends on pip and cython however it does not check the versions 
> of these.
> Some of the pip flags does not exist in older versions:
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/28#issuecomment-236382953
> (Note: Even though the above issue was reported by the user in a different 
> repo it is related to the apache beam sdk)
> Similarly with cython, SDK supports running with or without Cython. Because 
> of that reason it is not list it as a requirement in the setup.py file. 
> However, with an old version of cython SDK might fail.
> To avoid the above problem: In the SDK check the version of these packages 
> and show a warning to upgrade.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1007: Add support for bz2 compression in python

[tim1357]

Github user tim1357 closed the pull request at:

https://github.com/apache/incubator-beam/pull/1007


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-680) Python Dataflow stages stale requirements-cache dependencies

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524090#comment-15524090
 ] 

Scott Wegner commented on BEAM-680:
---

/cc [~robertwb]

This came up as an issue with 
[dependency_test.test_with_requirements_file()|https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/utils/dependency_test.py#L108]
 in [PR 1005|https://github.com/apache/incubator-beam/pull/1005].

We use pip to download all required dependencies, but generate the full listing 
by scanning the cache directory. Perhaps there is a way to ask pip for the 
transitive dependency list as well.

> Python Dataflow stages stale requirements-cache dependencies
> 
>
> Key: BEAM-680
> URL: https://issues.apache.org/jira/browse/BEAM-680
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Scott Wegner
>Priority: Minor
>
> When executing a python pipeline using a requirements.txt file, the Dataflow 
> runner will stage all dependencies downloaded to its requirements cache 
> directory, including those specified in the requirements.txt, and any 
> previously cached dependencies. This results in bloated staging directory if 
> previous pipeline runs from the same machine included different dependencies.
> Repro:
> # Initialize a virtualenv and pip install apache_beam
> # Create an empty requirements.txt file
> # Create a simple pipeline using DataflowPipelineRunner and a 
> requirements.txt file, for example: 
> [my_pipeline.py|https://gist.github.com/swegner/6df00df1423b48206c4ab5a7e917218a]
> # {{touch /tmp/dataflow-requirements-cache/extra-file.txt}}
> # Run the pipeline with a specified staging directory
> # Check the staged files for the job
> 'extra-file.txt' will be uploaded with the job, along with any other cached 
> dependencies under /tmp/dataflow-requirements-cache.
> We should only be staging the dependencies necessary for a pipeline, not all 
> previously-cached dependencies found on the machine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-680) Python Dataflow stages stale requirements-cache dependencies

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-680:
-

 Summary: Python Dataflow stages stale requirements-cache 
dependencies
 Key: BEAM-680
 URL: https://issues.apache.org/jira/browse/BEAM-680
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Scott Wegner
Priority: Minor


When executing a python pipeline using a requirements.txt file, the Dataflow 
runner will stage all dependencies downloaded to its requirements cache 
directory, including those specified in the requirements.txt, and any 
previously cached dependencies. This results in bloated staging directory if 
previous pipeline runs from the same machine included different dependencies.

Repro:

# Initialize a virtualenv and pip install apache_beam
# Create an empty requirements.txt file
# Create a simple pipeline using DataflowPipelineRunner and a requirements.txt 
file, for example: 
[my_pipeline.py|https://gist.github.com/swegner/6df00df1423b48206c4ab5a7e917218a]
# {{touch /tmp/dataflow-requirements-cache/extra-file.txt}}
# Run the pipeline with a specified staging directory
# Check the staged files for the job

'extra-file.txt' will be uploaded with the job, along with any other cached 
dependencies under /tmp/dataflow-requirements-cache.

We should only be staging the dependencies necessary for a pipeline, not all 
previously-cached dependencies found on the machine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Correctly type collections in PipelineOptions.

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 3fe3bc8eb -> 21c13d5f6


Correctly type collections in PipelineOptions.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4e9987c4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4e9987c4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4e9987c4

Branch: refs/heads/master
Commit: 4e9987c423f8934f802b44f3e82a6200978bdc37
Parents: 3fe3bc8
Author: sammcveety 
Authored: Thu Aug 18 20:49:05 2016 -0400
Committer: Dan Halperin 
Committed: Mon Sep 26 14:25:08 2016 -0700

--
 .../sdk/options/PipelineOptionsFactory.java |  92 ---
 .../sdk/options/PipelineOptionsFactoryTest.java | 111 +--
 2 files changed, 156 insertions(+), 47 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4e9987c4/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
index 43927bc..9fc6c2c 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
@@ -28,8 +28,8 @@ import com.google.common.base.Function;
 import com.google.common.base.Joiner;
 import com.google.common.base.Optional;
 import com.google.common.base.Predicate;
+import com.google.common.base.Predicates;
 import com.google.common.base.Strings;
-import com.google.common.collect.Collections2;
 import com.google.common.collect.FluentIterable;
 import com.google.common.collect.ImmutableListMultimap;
 import com.google.common.collect.ImmutableMap;
@@ -55,6 +55,7 @@ import java.lang.annotation.Annotation;
 import java.lang.reflect.Method;
 import java.lang.reflect.Modifier;
 import java.lang.reflect.Proxy;
+import java.lang.reflect.Type;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collection;
@@ -843,13 +844,8 @@ public class PipelineOptionsFactory {
* resolved.
*/
   private static List getPropertyDescriptors(
-  Class beanClass)
+Set methods, Class beanClass)
   throws IntrospectionException {
-// The sorting is important to make this method stable.
-SortedSet methods = Sets.newTreeSet(MethodComparator.INSTANCE);
-methods.addAll(
-Collections2.filter(Arrays.asList(beanClass.getMethods()), 
NOT_SYNTHETIC_PREDICATE));
-
 SortedMap propertyNamesToGetters = new TreeMap<>();
 for (Map.Entry entry :
 PipelineOptionsReflector.getPropertyNamesToGetters(methods).entries()) 
{
@@ -858,6 +854,7 @@ public class PipelineOptionsFactory {
 
 List descriptors = Lists.newArrayList();
 List mismatches = new ArrayList<>();
+Set usedDescriptors = Sets.newHashSet();
 /*
  * Add all the getter/setter pairs to the list of descriptors removing the 
getter once
  * it has been paired up.
@@ -874,9 +871,9 @@ public class PipelineOptionsFactory {
 
   // Validate that the getter and setter property types are the same.
   if (getterMethod != null) {
-Class getterPropertyType = getterMethod.getReturnType();
-Class setterPropertyType = method.getParameterTypes()[0];
-if (getterPropertyType != setterPropertyType) {
+Type getterPropertyType = getterMethod.getGenericReturnType();
+Type setterPropertyType = method.getGenericParameterTypes()[0];
+if (!getterPropertyType.equals(setterPropertyType)) {
   TypeMismatch mismatch = new TypeMismatch();
   mismatch.propertyName = propertyName;
   mismatch.getterPropertyType = getterPropertyType;
@@ -885,9 +882,14 @@ public class PipelineOptionsFactory {
   continue;
 }
   }
-
-  descriptors.add(new PropertyDescriptor(
-  propertyName, getterMethod, method));
+  // Properties can appear multiple times with subclasses, and we don't
+  // want to add a bad entry if we have already added a good one (with both
+  // getter and setter).
+  if (!usedDescriptors.contains(propertyName)) {
+descriptors.add(new PropertyDescriptor(
+propertyName, getterMethod, method));
+usedDescriptors.add(propertyName);
+  }
 }
 throwForTypeMismatches(mismatches);
 
@@ -901,8 +903,8 @@ public class PipelineOptionsFactory {
 
   private static class TypeMismatch {
 private String propertyName;
-private Class getterPropertyType;
-private Class 

[2/2] incubator-beam git commit: Closes #852

[dhalperi]

Closes #852


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/21c13d5f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/21c13d5f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/21c13d5f

Branch: refs/heads/master
Commit: 21c13d5f6f6913bda2dcfb52157f2e36aa86bf8b
Parents: 3fe3bc8 4e9987c
Author: Dan Halperin 
Authored: Mon Sep 26 14:25:31 2016 -0700
Committer: Dan Halperin 
Committed: Mon Sep 26 14:25:31 2016 -0700

--
 .../sdk/options/PipelineOptionsFactory.java |  92 ---
 .../sdk/options/PipelineOptionsFactoryTest.java | 111 +--
 2 files changed, 156 insertions(+), 47 deletions(-)
--

[GitHub] incubator-beam pull request #852: Correctly type collections in PipelineOpti...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/852


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Fix python bin test.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 7e744e445 -> 8a333a661


Fix python bin test.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4b4ad732
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4b4ad732
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4b4ad732

Branch: refs/heads/python-sdk
Commit: 4b4ad7328b5c9528f2aca2f2b8627ae0793cf1b6
Parents: 7e744e4
Author: Robert Bradshaw 
Authored: Mon Sep 26 14:14:03 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Sep 26 14:14:03 2016 -0700

--
 sdks/python/apache_beam/utils/dependency.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4b4ad732/sdks/python/apache_beam/utils/dependency.py
--
diff --git a/sdks/python/apache_beam/utils/dependency.py 
b/sdks/python/apache_beam/utils/dependency.py
index 150b2fc..314bd1b 100644
--- a/sdks/python/apache_beam/utils/dependency.py
+++ b/sdks/python/apache_beam/utils/dependency.py
@@ -193,7 +193,7 @@ def _get_python_executable():
   # Allow overriding the python executable to use for downloading and
   # installing dependencies, otherwise use the python executable for
   # the current process.
-  beam_python = os.environ.get('BEAM_PYTHON') or sys.executable
+  python_bin = os.environ.get('BEAM_PYTHON') or sys.executable
   if not python_bin:
 raise ValueError('Could not find Python executable.')
   return python_bin

[GitHub] incubator-beam pull request #1008: Fix python bin test.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1008


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1008

[robertwb]

Closes #1008


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8a333a66
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8a333a66
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8a333a66

Branch: refs/heads/python-sdk
Commit: 8a333a661ff7059f8f683954c81ec2ffbe7c5bcc
Parents: 7e744e4 4b4ad73
Author: Robert Bradshaw 
Authored: Mon Sep 26 14:36:25 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Sep 26 14:36:25 2016 -0700

--
 sdks/python/apache_beam/utils/dependency.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1210

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1006: [BEAM-678] Add BEAM_PYTHON environment ov...

[swegner]

Github user swegner closed the pull request at:

https://github.com/apache/incubator-beam/pull/1006


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #827: [BEAM-551] Add ValueProvider class

[sammcveety]

Github user sammcveety closed the pull request at:

https://github.com/apache/incubator-beam/pull/827


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1213

[Apache Jenkins Server]

See 

[2/2] incubator-beam git commit: Support BigQuery DATETIME and TIME types

[lcwik]

Support BigQuery DATETIME and TIME types

This closes #997


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/68010aba
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/68010aba
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/68010aba

Branch: refs/heads/master
Commit: 68010aba16f0660069d9eebdeb1ddf1f40d38414
Parents: 21c13d5 2bfc89e
Author: Luke Cwik 
Authored: Mon Sep 26 16:32:27 2016 -0700
Committer: Luke Cwik 
Committed: Mon Sep 26 16:32:27 2016 -0700

--
 .../sdk/io/gcp/bigquery/BigQueryAvroUtils.java  |  4 
 .../gcp/bigquery/BigQueryTableRowIterator.java  |  3 ++-
 .../io/gcp/bigquery/BigQueryAvroUtilsTest.java  | 20 +++-
 .../bigquery/BigQueryTableRowIteratorTest.java  | 20 ++--
 4 files changed, 35 insertions(+), 12 deletions(-)
--

[jira] [Created] (BEAM-682) Invoker Class should be created in Thread

[Sumit Chawla (JIRA)]

Sumit Chawla created BEAM-682:
-

 Summary: Invoker Class should be created in Thread
 Key: BEAM-682
 URL: https://issues.apache.org/jira/browse/BEAM-682
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: 0.3.0-incubating
Reporter: Sumit Chawla
Assignee: Sumit Chawla
Priority: Critical


As of now the InvokerClass is being loaded in wrong classloader. It should be 
loaded into Thread.currentThread.getContextClassLoader()


https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java#L167

{code}
 Class> res =
(Class>)
unloaded
.load(DoFnInvokers.class.getClassLoader(), 
ClassLoadingStrategy.Default.INJECTION)
.getLoaded();

{code}

Fix 
{code}
 Class> res =
(Class>)
unloaded
.load(Thread.currentThread().getContextClassLoader(),
ClassLoadingStrategy.Default.INJECTION)
.getLoaded();
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-682) Invoker Class should be created in Thread Context Classloader

[Sumit Chawla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sumit Chawla updated BEAM-682:
--
Summary: Invoker Class should be created in Thread Context Classloader  
(was: Invoker Class should be created in Thread)

> Invoker Class should be created in Thread Context Classloader
> -
>
> Key: BEAM-682
> URL: https://issues.apache.org/jira/browse/BEAM-682
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.3.0-incubating
>Reporter: Sumit Chawla
>Assignee: Sumit Chawla
>Priority: Critical
>
> As of now the InvokerClass is being loaded in wrong classloader. It should be 
> loaded into Thread.currentThread.getContextClassLoader()
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java#L167
> {code}
>  Class> res =
> (Class>)
> unloaded
> .load(DoFnInvokers.class.getClassLoader(), 
> ClassLoadingStrategy.Default.INJECTION)
> .getLoaded();
> {code}
> Fix 
> {code}
>  Class> res =
> (Class>)
> unloaded
> .load(Thread.currentThread().getContextClassLoader(),
> ClassLoadingStrategy.Default.INJECTION)
> .getLoaded();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1399

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1003: [BEAM-674] Source part of GridFS IO

[dkulp]

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/1003

[BEAM-674] Source part of GridFS IO

This is the "Source" part for GridFS based IO for beam.  (will work on Sink 
next, but would like to get this reviewed and merged first) . The "default" is 
to parse each file as text files (by line), but a parser function can be 
provided to take the InputStream and parse via whatever is required.   

For runners that can split into bundles it attempts to assign files in the 
grid to different bundles.  


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam gridfs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1003.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1003


commit d5cdc2429622f65a762774de8b5baf15334e55e2
Author: Daniel Kulp 
Date:   2016-09-16T20:58:56Z

Add GridFS io

commit a9212662744c14f10cd811540c3e9268c32c25c4
Author: Daniel Kulp 
Date:   2016-09-16T21:19:50Z

Fix checkstyle issues

commit cee0a06b6a465a276c2c5410d7d3f9af703982d4
Author: Daniel Kulp 
Date:   2016-09-19T17:22:44Z

Attempt to get a converter in there

commit fafa8fa607f22eacf918abb13419f28df9d2a8e9
Author: Daniel Kulp 
Date:   2016-09-19T17:32:44Z

Fix javac compile problem

commit 7e9872f12c74902f1a23e5a27eb0027ae753947a
Author: Daniel Kulp 
Date:   2016-09-19T17:50:11Z

Force a serializable

commit 265747946864b226235ee5b758e6c10b7cc3992f
Author: Daniel Kulp 
Date:   2016-09-19T17:56:03Z

Add the needed coder

commit 4f54495afe7ff4768d873350c345d39905d812fc
Author: Daniel Kulp 
Date:   2016-09-19T18:02:39Z

Change to using the GridFSDBFile instead of InputStream so the parsingFn 
can have access to tall the metadata

commit cbeebf02542a5e5a5f4b9a6c370b1b68b46d2deb
Author: Daniel Kulp 
Date:   2016-09-19T18:25:23Z

Flip to allowing the parser to have complete control over how the item is 
added to the collection

commit a08007b9f444fedcde78ab38c6cdf505b3864c61
Author: Daniel Kulp 
Date:   2016-09-19T18:26:19Z

Fix unused imports

commit a4840e98d891d3fa783654a472af06c4d399a929
Author: Daniel Kulp 
Date:   2016-09-21T19:51:45Z

Add test for the parser functionality and cleanup some of that code

commit 438a792a796be77186d79aa3fdb221efcced6d4f
Author: Daniel Kulp 
Date:   2016-09-21T20:01:33Z

Move the coder out from the parser

commit e8fcdbf3cebd6fa4648f328484dee07fec35b21a
Author: Daniel Kulp 
Date:   2016-09-22T12:36:50Z

Fix test

commit 1d1a373fc7cec4e78bf0e618a902c15005fc36b4
Author: Daniel Kulp 
Date:   2016-09-23T14:49:53Z

Flip to using BoundedSource so it can be broken up into bundles




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-676) Use custom POJO and create JMS ConnectionFactory

[JIRA]

Jean-Baptiste Onofré created BEAM-676:
-

 Summary: Use custom POJO and create JMS ConnectionFactory
 Key: BEAM-676
 URL: https://issues.apache.org/jira/browse/BEAM-676
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


The {{JmsIO}} uses JMS {{ConnectionFactory}} as configuration. Unfortunately, 
{{ConnectionFactory}} interface doesn't extend {{Serializable}} (even if most 
of the implementations are {{Serializable}}).
Instead (as I'm doing for the {{JdbcIO}}), I propose to create a 
{{ConnectionFactoryConfiguration}} POJO to create the {{ConnectionFactory}} in 
the {{@Setup}} of the reader and writer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-559) DoFnTester should handle Setup / TearDown

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-559:
-
Assignee: Eugene Kirpichov  (was: Thomas Groh)

> DoFnTester should handle Setup / TearDown
> -
>
> Key: BEAM-559
> URL: https://issues.apache.org/jira/browse/BEAM-559
> Project: Beam
>  Issue Type: Improvement
>Reporter: Vikas Kedigehalli
>Assignee: Eugene Kirpichov
>
> Now that DoFn supports setup and teardown, it would be nice for DoFnTester to 
> add them to its lifecycle so as to avoid calling these methods explicitly in 
> DoFn unit tests. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1215

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1214

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1011: Support for @Setup and @Teardown in DoFnT...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/1011

Support for @Setup and @Teardown in DoFnTester

- To allow testing DoFn reuse, adds a third cloning mode - CLONE_ONCE. The 
current CLONE is renamed to CLONE_PER_BUNDLE because it corresponds to the 
olden ways of DoFnTester. This rename should be harmless because it was the 
default, so probably nobody uses it (nobody uses it in the Beam codebase 
anyway), but I can rename it back if necessary.
- Calls Setup lazily on the first interaction with the fn under test: 
depending on cloning mode, this is either a per-bundle clone, or the original 
fn, or a single clone of the original fn.
- Calls Teardown either when the DoFnTester is closed, or when 
finishBundle'ing with CLONE_PER_BUNDLE.
- Enforces that you have to configure the DoFnTester (setSideInputs, 
setCloningBehavior) before using it, to simplify the semantics. Makes 
setSideInputs consistent with setSideInput(without "s") in not resetting the 
state.
- Allows processing a bundle after processing an element. This used to be 
disallowed, but it's weird, because processing a bundle after processing a 
bundle *was* allowed.

Additionally, improves testing coverage.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam do-fn-tester-lifecycle

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1011.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1011


commit 62ec3ed9bfa0bb94c41b0df4761d5d50cb4d196a
Author: Eugene Kirpichov 
Date:   2016-09-26T23:58:20Z

Support for @Setup and @Teardown in DoFnTester




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-683) Make BZIP compressed files splittable

[Tim Sears (JIRA)]

Tim Sears created BEAM-683:
--

 Summary: Make BZIP compressed files splittable 
 Key: BEAM-683
 URL: https://issues.apache.org/jira/browse/BEAM-683
 Project: Beam
  Issue Type: Wish
Reporter: Tim Sears
Priority: Minor


Bzip2 is compressed as blocks, so it should be possible to do dynamic 
splitting. To do this: Seek to a location in the bzip, then keep seeking until 
you find the 6 byte block-start sequence 0x314159265359 (which is the 12 digit 
approximation of pi). You can use a bzip2 decompressor from that point onwards.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-559) DoFnTester should handle Setup / TearDown

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524677#comment-15524677
 ] 

Eugene Kirpichov commented on BEAM-559:
---

PR https://github.com/apache/incubator-beam/pull/1011

> DoFnTester should handle Setup / TearDown
> -
>
> Key: BEAM-559
> URL: https://issues.apache.org/jira/browse/BEAM-559
> Project: Beam
>  Issue Type: Improvement
>Reporter: Vikas Kedigehalli
>Assignee: Eugene Kirpichov
>
> Now that DoFn supports setup and teardown, it would be nice for DoFnTester to 
> add them to its lifecycle so as to avoid calling these methods explicitly in 
> DoFn unit tests. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1395

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-517) Check versions of pip and cython

[Marco Buccini (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522607#comment-15522607
 ] 

Marco Buccini commented on BEAM-517:


So, from my understanding we could specify the requirements for PIP in setup.py 
with the following:

import pkg_resources
assert str(pkg_resources.get_distribution('pip').version) >= '7.0.0', "This SDK 
requires 'pip' >= 7.0.0"

while mentioning it in the README file. 
(it seems that the --no-binary flag was introduced in version 7.0.0: 
https://pip.pypa.io/en/stable/news/ )

Regarding cython, I am not quite sure about the version needed to run the code, 
so probably someone else might be able to tell more about it.

> Check versions of pip and cython
> 
>
> Key: BEAM-517
> URL: https://issues.apache.org/jira/browse/BEAM-517
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> Python SDK depends on pip and cython however it does not check the versions 
> of these.
> Some of the pip flags does not exist in older versions:
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/28#issuecomment-236382953
> (Note: Even though the above issue was reported by the user in a different 
> repo it is related to the apache beam sdk)
> Similarly with cython, SDK supports running with or without Cython. Because 
> of that reason it is not list it as a requirement in the setup.py file. 
> However, with an old version of cython SDK might fail.
> To avoid the above problem: In the SDK check the version of these packages 
> and show a warning to upgrade.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-644) Primitive to shift the watermark while assigning timestamps

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522966#comment-15522966
 ] 

Aljoscha Krettek commented on BEAM-644:
---

[~kenn] I was referring to "two clusters of elements from two separate input 
elements" but that's somewhat besides the point because I was thinking about 
how a Kafka source would be implemented as a combination of {{DoFn}} plus 
{{SplittableDoFn}}. There you need to manage the watermark at the 
{{SplittableDoFn}} which would be responsible for reading from topics.

I think we might be talking about different things here. As I said, the 
proposed changes are very good in how they simplify the API of {{DoFn}} and 
also clean up stuff around allowed time skew.

What I was thinking about is in general a problem with watermarks. I though 
that the proposal here was meant to fixed that but I don't think we can. What I 
was trying to get at essentially boils down to this: If we want our watermark 
to be 100 % correct then we can never advance it because we never know what 
timestamps future elements will have. (For the general case, where any data 
with any timestamp can arrive at any point in (processing) time.). I was just 
pondering that and I'm afraid it derailed the discussion a bit.

> Primitive to shift the watermark while assigning timestamps
> ---
>
> Key: BEAM-644
> URL: https://issues.apache.org/jira/browse/BEAM-644
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> There is a general need, especially important in the presence of 
> SplittableDoFn, to be able to assign new timestamps to elements without 
> making them late or droppable.
>  - DoFn.withAllowedTimestampSkew is inadequate, because it simply allows one 
> to produce late data, but does not allow one to shift the watermark so the 
> new data is on-time.
>  - For a SplittableDoFn, one may receive an element such as the name of a log 
> file that contains elements for the day preceding the log file. The timestamp 
> on the filename must currently be the beginning of the log. If such elements 
> are constantly flowing, it may be OK, but since we don't know that element is 
> coming, in that absence of data, the watermark may advance. We need a way to 
> keep it far enough back even in the absence of data holding it back.
> One idea is a new primitive ShiftWatermark / AdjustTimestamps with the 
> following pieces:
>  - A constant duration (positive or negative) D by which to shift the 
> watermark.
>  - A function from TimestampedElement to new timestamp that is >= t + D
> So, for example, AdjustTimestamps(<-60 minutes>, f) would allow f to make 
> timestamps up to 60 minutes earlier.
> With this primitive added, outputWithTimestamp and withAllowedTimestampSkew 
> could be removed, simplifying DoFn.
> Alternatively, all of this functionality could be bolted on to DoFn.
> This ticket is not a proposal, but a record of the issue and ideas that were 
> mentioned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1211

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-673) Data locality for Read.Bounded

[Amit Sela (JIRA)]

Amit Sela created BEAM-673:
--

 Summary: Data locality for Read.Bounded
 Key: BEAM-673
 URL: https://issues.apache.org/jira/browse/BEAM-673
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Amit Sela
Assignee: Amit Sela


In some distributed filesystems, such as HDFS, we should be able to hint to 
Spark the preferred locations of splits.
Here is an example of how Spark does that for Hadoop RDDs:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala#L252

*Note: in case of 1-to-1 mapping of Read operation (e.g. TextIO) direct 
translation should still be preferred, but this is pending HDFS support for 
Beam anyway.*



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1396

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1396

[Apache Jenkins Server]

See 

[GitHub] incubator-beam-site pull request #40: Create the WordCount walkthrough [WIP]

[hadarhg]

GitHub user hadarhg opened a pull request:

https://github.com/apache/incubator-beam-site/pull/40

Create the WordCount walkthrough [WIP]

Create the WordCount Walkthrough:
[x] - Minimal WordCount section
[x] - WordCount section
[x] - Debugging WordCount section
[] - Windowing WordCount section
[] - Include code snippets from real code
[] - Directions for how to run each example

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hadarhg/incubator-beam-site word-count

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/40.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #40


commit 4e6a16932e7f945ea6977cc3d508b930ad0cf582
Author: Hadar Hod 
Date:   2016-09-26T19:31:16Z

Create WordCount walkthrough. This is a work in progress.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-684) De-Dataflow Beam

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525156#comment-15525156
 ] 

ASF GitHub Bot commented on BEAM-684:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1016

[BEAM-684] pom.xml: replace dataflow.javadoc_opts with beam.javadoc_opts



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam 
fix-dataflow-references

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1016.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1016


commit 74e5b7eba7fabe169d8955ecfb6c9e304ddcc442
Author: Dan Halperin 
Date:   2016-09-27T05:43:40Z

pom.xml: replace dataflow.javadoc_opts with beam.javadoc_opts




> De-Dataflow Beam
> 
>
> Key: BEAM-684
> URL: https://issues.apache.org/jira/browse/BEAM-684
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
>
> {code}
> grep -i dataflow sdks/java/core/**/*.java | wc -l
>  136
> {code}
> Most of those uses of the word {{dataflow}} are wrong. We should really 
> finish converting over to Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-475) High-quality javadoc for Beam

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525122#comment-15525122
 ] 

ASF GitHub Bot commented on BEAM-475:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1015

[BEAM-475] pom.xml: produce aggregate javadoc for the project



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam javadoc-aggregating

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1015.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1015


commit d52408fdaf467d816bf6ae1ac0fcc6d09fa6c450
Author: Dan Halperin 
Date:   2016-09-27T05:13:20Z

pom.xml: produce aggregate javadoc for the project




> High-quality javadoc for Beam
> -
>
> Key: BEAM-475
> URL: https://issues.apache.org/jira/browse/BEAM-475
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> We should have good Javadoc for Beam!
> Current snapshot: http://beam.incubator.apache.org/javadoc/0.1.0-incubating/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1015: [BEAM-475] pom.xml: produce aggregate jav...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1015

[BEAM-475] pom.xml: produce aggregate javadoc for the project



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam javadoc-aggregating

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1015.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1015


commit d52408fdaf467d816bf6ae1ac0fcc6d09fa6c450
Author: Dan Halperin 
Date:   2016-09-27T05:13:20Z

pom.xml: produce aggregate javadoc for the project




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-684) De-Dataflow Beam

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-684:


 Summary: De-Dataflow Beam
 Key: BEAM-684
 URL: https://issues.apache.org/jira/browse/BEAM-684
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: Daniel Halperin


{code}
grep -i dataflow sdks/java/core/**/*.java | wc -l
 136
{code}

Most of those uses of the word {{dataflow}} are wrong. We should really finish 
converting over to Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-684) De-Dataflow Beam

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-684:
-
Priority: Minor  (was: Major)

> De-Dataflow Beam
> 
>
> Key: BEAM-684
> URL: https://issues.apache.org/jira/browse/BEAM-684
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
>
> {code}
> grep -i dataflow sdks/java/core/**/*.java | wc -l
>  136
> {code}
> Most of those uses of the word {{dataflow}} are wrong. We should really 
> finish converting over to Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1016: [BEAM-684] pom.xml: replace dataflow.java...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1016

[BEAM-684] pom.xml: replace dataflow.javadoc_opts with beam.javadoc_opts



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam 
fix-dataflow-references

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1016.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1016


commit 74e5b7eba7fabe169d8955ecfb6c9e304ddcc442
Author: Dan Halperin 
Date:   2016-09-27T05:43:40Z

pom.xml: replace dataflow.javadoc_opts with beam.javadoc_opts




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-684) De-Dataflow Beam

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525164#comment-15525164
 ] 

Jean-Baptiste Onofré commented on BEAM-684:
---

Big +1

> De-Dataflow Beam
> 
>
> Key: BEAM-684
> URL: https://issues.apache.org/jira/browse/BEAM-684
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
>
> {code}
> grep -i dataflow sdks/java/core/**/*.java | wc -l
>  136
> {code}
> Most of those uses of the word {{dataflow}} are wrong. We should really 
> finish converting over to Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1012: Update Dataflow worker image for Apache B...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/1012

Update Dataflow worker image for Apache Beam

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam 
update_dataflow_vm_image

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1012.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1012


commit fad30d9b9b5c11098b61e8f73c1a5cfbdaa0bd85
Author: Luke Cwik 
Date:   2016-09-27T02:25:39Z

Update Dataflow worker image for Apache Beam




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1013: BEAM-682-Invoker Class should be created ...

[sumitchawla]

GitHub user sumitchawla opened a pull request:

https://github.com/apache/incubator-beam/pull/1013

BEAM-682-Invoker Class should be created in Thread Context Classloader

As of now the InvokerClass is being loaded in wrong classloader. It should 
be loaded into Thread.currentThread.getContextClassLoader()

https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java#L167
 Class> res =
(Class>)
unloaded
.load(DoFnInvokers.class.getClassLoader(), 
ClassLoadingStrategy.Default.INJECTION)
.getLoaded();

Fix
 Class> res =
(Class>)
unloaded
.load(Thread.currentThread().getContextClassLoader(),
ClassLoadingStrategy.Default.INJECTION)
.getLoaded();

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sumitchawla/incubator-beam classloader_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1013.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1013


commit ecb7fc6c7ec7b0a2016f7da5edf9cf7f3bc9ace4
Author: Sumit Chawla 
Date:   2016-09-27T02:45:22Z

BEAM-682-Invoker Class should be created in Thread Context Classloader




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[44/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/index.html
--
diff --git a/content/learn/sdks/javadoc/0.2.0-incubating/index.html 
b/content/learn/sdks/javadoc/0.2.0-incubating/index.html
new file mode 100644
index 000..03b91f7
--- /dev/null
+++ b/content/learn/sdks/javadoc/0.2.0-incubating/index.html
@@ -0,0 +1,75 @@
+http://www.w3.org/TR/html4/frameset.dtd;>
+
+
+
+
+
+Apache Beam SDK for Java, version 0.2.0-incubating
+
+targetPage = "" + window.location.search;
+if (targetPage != "" && targetPage != "undefined")
+targetPage = targetPage.substring(1);
+if (targetPage.indexOf(":") != -1 || (targetPage != "" && 
!validURL(targetPage)))
+targetPage = "undefined";
+function validURL(url) {
+try {
+url = decodeURIComponent(url);
+}
+catch (error) {
+return false;
+}
+var pos = url.indexOf(".html");
+if (pos == -1 || pos != url.length - 5)
+return false;
+var allowNumber = false;
+var allowSep = false;
+var seenDot = false;
+for (var i = 0; i < url.length - 5; i++) {
+var ch = url.charAt(i);
+if ('a' <= ch && ch <= 'z' ||
+'A' <= ch && ch <= 'Z' ||
+ch == '$' ||
+ch == '_' ||
+ch.charCodeAt(0) > 127) {
+allowNumber = true;
+allowSep = true;
+} else if ('0' <= ch && ch <= '9'
+|| ch == '-') {
+if (!allowNumber)
+ return false;
+} else if (ch == '/' || ch == '.') {
+if (!allowSep)
+return false;
+allowNumber = false;
+allowSep = false;
+if (ch == '.')
+ seenDot = true;
+if (ch == '/' && seenDot)
+ return false;
+} else {
+return false;
+}
+}
+return true;
+}
+function loadFrames() {
+if (targetPage != "" && targetPage != "undefined")
+ top.classFrame.location = top.targetPage;
+}
+
+
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+Frame Alert
+This document is designed to be viewed using the frames feature. If you see 
this message, you are using a non-frame-capable web client. Link to Non-frame version.
+
+
+

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/javadoc.sh
--
diff --git a/content/learn/sdks/javadoc/0.2.0-incubating/javadoc.sh 
b/content/learn/sdks/javadoc/0.2.0-incubating/javadoc.sh
new file mode 100755
index 000..e0b0acd
--- /dev/null
+++ b/content/learn/sdks/javadoc/0.2.0-incubating/javadoc.sh
@@ -0,0 +1 @@
+/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/../bin/javadoc
 @options @packages
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/options
--
diff --git a/content/learn/sdks/javadoc/0.2.0-incubating/options 
b/content/learn/sdks/javadoc/0.2.0-incubating/options
new file mode 100644
index 000..3cf35bc
--- /dev/null
+++ b/content/learn/sdks/javadoc/0.2.0-incubating/options
@@ -0,0 +1,24 @@
+-classpath
+'/Users/dhalperi/IdeaProjects/incubator-beam/sdks/target/classes:/Users/dhalperi/IdeaProjects/incubator-beam/sdks/java/target/classes:/Users/dhalperi/IdeaProjects/incubator-beam/sdks/java/build-tools/target/classes:/Users/dhalperi/IdeaProjects/incubator-beam/sdks/java/core/target/classes:/Users/dhalperi/IdeaProjects/incubator-beam/runners/target/classes:/Users/dhalperi/IdeaProjects/incubator-beam/runners/core-java/target/classes:/Users/dhalperi/IdeaProjects/incubator-beam/runners/direct-java/target/classes:/Users/dhalperi/IdeaProjects/incubator-beam/runners/google-cloud-dataflow-java/target/classes:/Users/dhalperi/IdeaProjects/incubator-beam/sdks/java/io/target/classes:/Users/dhalperi/IdeaProjects/incubator-beam/sdks/java/io/google-cloud-platform/target/classes:/Users/dhalperi/IdeaProjects/incubator-beam/sdks/java/io/hdfs/target/classes:/Users/dhalperi/IdeaProjects/incubator-beam/sdks/java/io/jms/target/classes:/Users/dhalperi/IdeaProjects/incubator-beam/sdks/java/io/kafka/target/cl
 

[39/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.TransformTranslator.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.TransformTranslator.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.TransformTranslator.html
new file mode 100644
index 000..9d384e0
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.TransformTranslator.html
@@ -0,0 +1,235 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+DataflowPipelineTranslator.TransformTranslator (Apache Beam SDK for 
Java, version 0.2.0-incubating)
+
+
+
+
+
+
+var methods = {"i0":6};
+var tabs = {65535:["t0","All Methods"],2:["t2","Instance 
Methods"],4:["t3","Abstract Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.dataflow
+Interface 
DataflowPipelineTranslator.TransformTranslatorTransformT extends PTransform
+
+
+
+
+
+
+Enclosing class:
+DataflowPipelineTranslator
+
+
+
+public static interface DataflowPipelineTranslator.TransformTranslatorTransformT
 extends PTransform
+A DataflowPipelineTranslator.TransformTranslator
 knows how to translate
+ a particular subclass of PTransform for 
the
+ Cloud Dataflow service. It does so by
+ mutating the DataflowPipelineTranslator.TranslationContext.
+
+
+
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsInstance MethodsAbstract Methods
+
+Modifier and Type
+Method and Description
+
+
+void
+translate(TransformTtransform,
+ DataflowPipelineTranslator.TranslationContextcontext)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Method Detail
+
+
+
+
+
+
+
+translate
+voidtranslate(TransformTtransform,
+   DataflowPipelineTranslator.TranslationContextcontext)
+
+
+
+
+
+
+
+
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+Copyright  2016 http://www.apache.org/;>The Apache Software Foundation. All rights 
reserved.
+
+

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.TranslationContext.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.TranslationContext.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.TranslationContext.html
new file mode 100644
index 000..907380a
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.TranslationContext.html
@@ -0,0 +1,548 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+DataflowPipelineTranslator.TranslationContext (Apache Beam SDK for 
Java, version 0.2.0-incubating)
+
+
+
+
+
+
+var methods = 
{"i0":6,"i1":6,"i2":6,"i3":6,"i4":6,"i5":6,"i6":6,"i7":6,"i8":6,"i9":6,"i10":6,"i11":6,"i12":6,"i13":6,"i14":6,"i15":6,"i16":6};
+var tabs = {65535:["t0","All Methods"],2:["t2","Instance 
Methods"],4:["t3","Abstract Methods"]};
+var altColor = "altColor";
+var rowColor = 

[18/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/MonitoringUtil.JobMessagesHandler.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/MonitoringUtil.JobMessagesHandler.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/MonitoringUtil.JobMessagesHandler.html
new file mode 100644
index 000..7ec8e49
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/MonitoringUtil.JobMessagesHandler.html
@@ -0,0 +1,198 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Interface 
org.apache.beam.runners.dataflow.util.MonitoringUtil.JobMessagesHandler (Apache 
Beam SDK for Java, version 0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+
+Uses of 
Interfaceorg.apache.beam.runners.dataflow.util.MonitoringUtil.JobMessagesHandler
+
+
+
+
+
+Packages that use MonitoringUtil.JobMessagesHandler
+
+Package
+Description
+
+
+
+org.apache.beam.runners.dataflow
+
+Provides a Beam runner that executes pipelines on the 
Google Cloud Dataflow service.
+
+
+
+org.apache.beam.runners.dataflow.util
+
+Provides miscellaneous internal utilities used by the 
Google Cloud Dataflow runner.
+
+
+
+
+
+
+
+
+
+
+Uses of MonitoringUtil.JobMessagesHandler in 
org.apache.beam.runners.dataflow
+
+Methods in org.apache.beam.runners.dataflow
 with parameters of type MonitoringUtil.JobMessagesHandler
+
+Modifier and Type
+Method and Description
+
+
+
+PipelineResult.State
+DataflowPipelineJob.waitToFinish(longtimeToWait,
+http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/TimeUnit.html?is-external=true;
 title="class or interface in java.util.concurrent">TimeUnittimeUnit,
+MonitoringUtil.JobMessagesHandlermessageHandler)
+Waits for the job to finish and return the final 
status.
+
+
+
+
+
+
+
+
+Uses of MonitoringUtil.JobMessagesHandler in 
org.apache.beam.runners.dataflow.util
+
+Classes in org.apache.beam.runners.dataflow.util
 that implement MonitoringUtil.JobMessagesHandler
+
+Modifier and Type
+Class and Description
+
+
+
+static class
+MonitoringUtil.LoggingHandler
+A handler that logs monitoring messages.
+
+
+
+
+
+
+
+
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+Copyright  2016 http://www.apache.org/;>The Apache Software Foundation. All rights 
reserved.
+
+

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/MonitoringUtil.LoggingHandler.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/MonitoringUtil.LoggingHandler.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/MonitoringUtil.LoggingHandler.html
new file mode 100644
index 000..ad41773
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/MonitoringUtil.LoggingHandler.html
@@ -0,0 +1,126 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Class 
org.apache.beam.runners.dataflow.util.MonitoringUtil.LoggingHandler (Apache 
Beam SDK for Java, version 0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview

[51/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

Regenerated html


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/65d23415
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/65d23415
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/65d23415

Branch: refs/heads/asf-site
Commit: 65d23415255d53eb7711826d3483fb3daeb36b66
Parents: 125c117
Author: Frances Perry 
Authored: Mon Sep 26 20:58:58 2016 -0700
Committer: Frances Perry 
Committed: Mon Sep 26 20:58:58 2016 -0700

--
 content/feed.xml| 4 +-
 .../learn/runners/capability-matrix/index.html  | 2 +-
 .../0.2.0-incubating/allclasses-frame.html  |   777 +
 .../0.2.0-incubating/allclasses-noframe.html|   777 +
 .../0.2.0-incubating/constant-values.html   |  1251 ++
 .../0.2.0-incubating/deprecated-list.html   |   368 +
 .../sdks/javadoc/0.2.0-incubating/help-doc.html |   231 +
 .../javadoc/0.2.0-incubating/index-all.html | 17910 +
 .../sdks/javadoc/0.2.0-incubating/index.html|75 +
 .../sdks/javadoc/0.2.0-incubating/javadoc.sh| 1 +
 .../learn/sdks/javadoc/0.2.0-incubating/options |24 +
 .../core/GroupAlsoByWindowViaWindowSetDoFn.html |   413 +
 ...dSource.BoundedToUnboundedSourceAdapter.html |   450 +
 .../core/UnboundedReadFromBoundedSource.html|   452 +
 .../GroupAlsoByWindowViaWindowSetDoFn.html  |   126 +
 ...dSource.BoundedToUnboundedSourceAdapter.html |   169 +
 .../UnboundedReadFromBoundedSource.html |   126 +
 .../apache/beam/runners/core/package-frame.html |23 +
 .../beam/runners/core/package-summary.html  |   167 +
 .../apache/beam/runners/core/package-tree.html  |   157 +
 .../apache/beam/runners/core/package-use.html   |   163 +
 .../dataflow/BlockingDataflowRunner.html|   386 +
 .../DataflowJobAlreadyExistsException.html  |   290 +
 .../DataflowJobAlreadyUpdatedException.html |   289 +
 .../dataflow/DataflowJobCancelledException.html |   306 +
 .../runners/dataflow/DataflowJobException.html  |   277 +
 .../dataflow/DataflowJobExecutionException.html |   243 +
 .../dataflow/DataflowJobUpdatedException.html   |   347 +
 .../runners/dataflow/DataflowPipelineJob.html   |   461 +
 .../DataflowPipelineRegistrar.Options.html  |   288 +
 .../DataflowPipelineRegistrar.Runner.html   |   292 +
 .../dataflow/DataflowPipelineRegistrar.html |   228 +
 ...flowPipelineTranslator.JobSpecification.html |   301 +
 ...wPipelineTranslator.TransformTranslator.html |   235 +
 ...owPipelineTranslator.TranslationContext.html |   548 +
 .../dataflow/DataflowPipelineTranslator.html|   380 +
 .../beam/runners/dataflow/DataflowRunner.html   |   503 +
 .../runners/dataflow/DataflowRunnerHooks.html   |   287 +
 .../dataflow/DataflowServiceException.html  |   242 +
 .../class-use/BlockingDataflowRunner.html   |   170 +
 .../DataflowJobAlreadyExistsException.html  |   126 +
 .../DataflowJobAlreadyUpdatedException.html |   126 +
 .../DataflowJobCancelledException.html  |   126 +
 .../class-use/DataflowJobException.html |   203 +
 .../DataflowJobExecutionException.html  |   126 +
 .../class-use/DataflowJobUpdatedException.html  |   126 +
 .../dataflow/class-use/DataflowPipelineJob.html |   265 +
 .../DataflowPipelineRegistrar.Options.html  |   126 +
 .../DataflowPipelineRegistrar.Runner.html   |   126 +
 .../class-use/DataflowPipelineRegistrar.html|   126 +
 ...flowPipelineTranslator.JobSpecification.html |   172 +
 ...wPipelineTranslator.TransformTranslator.html |   189 +
 ...owPipelineTranslator.TranslationContext.html |   169 +
 .../class-use/DataflowPipelineTranslator.html   |   176 +
 .../dataflow/class-use/DataflowRunner.html  |   199 +
 .../dataflow/class-use/DataflowRunnerHooks.html |   176 +
 .../class-use/DataflowServiceException.html |   126 +
 .../BlockingDataflowPipelineOptions.html|   306 +
 .../dataflow/options/CloudDebuggerOptions.html  |   307 +
 ...elineDebugOptions.DataflowClientFactory.html |   295 +
 ...pelineDebugOptions.PathValidatorFactory.html |   296 +
 ...aflowPipelineDebugOptions.StagerFactory.html |   296 +
 .../options/DataflowPipelineDebugOptions.html   |   757 +
 .../DataflowPipelineOptions.JobNameFactory.html |   301 +
 ...wPipelineOptions.StagingLocationFactory.html |   295 +
 .../options/DataflowPipelineOptions.html|   476 +
 ...kerPoolOptions.AutoscalingAlgorithmType.html |   387 +
 ...ipelineWorkerPoolOptions.TeardownPolicy.html |   389 +
 ...ions.WorkerHarnessContainerImageFactory.html |   296 +
 .../DataflowPipelineWorkerPoolOptions.html  |   670 +
 ...ons.DataflowProfilingAgentConfiguration.html |   305 +
 .../options/DataflowProfilingOptions.html   |   291 +
 .../options/DataflowWorkerHarnessOptions.html   |   421 +
 

[19/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/RandomAccessData.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/RandomAccessData.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/RandomAccessData.html
new file mode 100644
index 000..578b324
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/RandomAccessData.html
@@ -0,0 +1,607 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+RandomAccessData (Apache Beam SDK for Java, version 
0.2.0-incubating)
+
+
+
+
+
+
+var methods = 
{"i0":10,"i1":10,"i2":10,"i3":10,"i4":10,"i5":10,"i6":10,"i7":10,"i8":10,"i9":10,"i10":10,"i11":10};
+var tabs = {65535:["t0","All Methods"],2:["t2","Instance 
Methods"],8:["t4","Concrete Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.dataflow.util
+Class RandomAccessData
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">java.lang.Object
+
+
+org.apache.beam.runners.dataflow.util.RandomAccessData
+
+
+
+
+
+
+
+
+http://docs.oracle.com/javase/7/docs/api/javax/annotation/concurrent.NotThreadSafe.html?is-external=true;
 title="class or interface in javax.annotation">@NotThreadSafe
+public class RandomAccessData
+extends http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">Object
+An elastic-sized byte array which allows you to manipulate 
it as a stream, or access
+ it directly. This allows for a quick succession of moving bytes from an http://docs.oracle.com/javase/7/docs/api/java/io/InputStream.html?is-external=true;
 title="class or interface in java.io">InputStream
+ to this wrapper to be used as an http://docs.oracle.com/javase/7/docs/api/java/io/OutputStream.html?is-external=true;
 title="class or interface in java.io">OutputStream and vice 
versa. This wrapper
+ also provides random access to bytes stored within. This wrapper allows users 
to finely
+ control the number of byte copies that occur.
+
+ Anything stored within the in-memory buffer from offset size()
 is considered temporary
+ unused storage.
+
+
+
+
+
+
+
+
+
+
+
+Nested Class Summary
+
+Nested Classes
+
+Modifier and Type
+Class and Description
+
+
+static class
+RandomAccessData.RandomAccessDataCoder
+A Coder which encodes the valid 
parts of this stream.
+
+
+
+static class
+RandomAccessData.UnsignedLexicographicalComparator
+A http://docs.oracle.com/javase/7/docs/api/java/util/Comparator.html?is-external=true;
 title="class or interface in java.util">Comparator that 
compares two byte arrays lexicographically.
+
+
+
+
+
+
+
+
+
+
+Field Summary
+
+Fields
+
+Modifier and Type
+Field and Description
+
+
+static RandomAccessData.UnsignedLexicographicalComparator
+UNSIGNED_LEXICOGRAPHICAL_COMPARATOR
+
+
+
+
+
+
+
+
+
+Constructor Summary
+
+Constructors
+
+Constructor and Description
+
+
+RandomAccessData()
+Constructs a RandomAccessData with a default buffer 
size.
+
+
+
+RandomAccessData(byte[]initialBuffer)
+Constructs a RandomAccessData with the initial buffer.
+
+
+
+RandomAccessData(intinitialBufferSize)
+Constructs a RandomAccessData with the given buffer 
size.
+
+
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsInstance MethodsConcrete Methods
+
+Modifier and Type
+Method and Description
+
+
+byte[]
+array()
+Returns the backing array.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/io/InputStream.html?is-external=true;
 title="class or interface in java.io">InputStream
+asInputStream(intoffset,
+ intlength)
+Returns an http://docs.oracle.com/javase/7/docs/api/java/io/InputStream.html?is-external=true;
 title="class or interface in java.io">InputStream wrapper 
which supplies the portion of this backing byte buffer
+ starting at offset and up to length bytes.
+
+
+

[32/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.TeardownPolicy.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.TeardownPolicy.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.TeardownPolicy.html
new file mode 100644
index 000..f87a083
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.TeardownPolicy.html
@@ -0,0 +1,389 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+DataflowPipelineWorkerPoolOptions.TeardownPolicy (Apache Beam SDK for 
Java, version 0.2.0-incubating)
+
+
+
+
+
+
+var methods = {"i0":10,"i1":9,"i2":9};
+var tabs = {65535:["t0","All Methods"],1:["t1","Static 
Methods"],2:["t2","Instance Methods"],8:["t4","Concrete Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Enum Constants|
+Field|
+Method
+
+
+Detail:
+Enum Constants|
+Field|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.dataflow.options
+Enum DataflowPipelineWorkerPoolOptions.TeardownPolicy
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">java.lang.Object
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Enum.html?is-external=true;
 title="class or interface in java.lang">java.lang.EnumDataflowPipelineWorkerPoolOptions.TeardownPolicy
+
+
+org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions.TeardownPolicy
+
+
+
+
+
+
+
+
+
+All Implemented Interfaces:
+http://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html?is-external=true;
 title="class or interface in java.io">Serializable, http://docs.oracle.com/javase/7/docs/api/java/lang/Comparable.html?is-external=true;
 title="class or interface in java.lang">ComparableDataflowPipelineWorkerPoolOptions.TeardownPolicy
+
+
+Enclosing interface:
+DataflowPipelineWorkerPoolOptions
+
+
+
+public static enum DataflowPipelineWorkerPoolOptions.TeardownPolicy
+extends http://docs.oracle.com/javase/7/docs/api/java/lang/Enum.html?is-external=true;
 title="class or interface in java.lang">EnumDataflowPipelineWorkerPoolOptions.TeardownPolicy
+The policy for tearing down the workers spun up by the 
service.
+
+
+
+
+
+
+
+
+
+
+
+Enum Constant Summary
+
+Enum Constants
+
+Enum Constant and Description
+
+
+TEARDOWN_ALWAYS
+All VMs created for a Dataflow job are deleted when the job 
finishes, regardless of whether
+ it fails or succeeds.
+
+
+
+TEARDOWN_NEVER
+All VMs created for a Dataflow job are left running when 
the job finishes, regardless of
+ whether it fails or succeeds.
+
+
+
+TEARDOWN_ON_SUCCESS
+All VMs created for a Dataflow job are deleted when the job 
succeeds, but are left running
+ when it fails.
+
+
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsStatic MethodsInstance MethodsConcrete Methods
+
+Modifier and Type
+Method and Description
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">String
+getTeardownPolicyName()
+
+
+static DataflowPipelineWorkerPoolOptions.TeardownPolicy
+valueOf(http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">Stringname)
+Returns the enum constant of this type with the specified 
name.
+
+
+
+static DataflowPipelineWorkerPoolOptions.TeardownPolicy[]
+values()
+Returns an array containing the constants of this enum 
type, in
+the order they are declared.
+
+
+
+
+
+
+
+Methods inherited from classjava.lang.http://docs.oracle.com/javase/7/docs/api/java/lang/Enum.html?is-external=true;
 title="class or interface in java.lang">Enum
+http://docs.oracle.com/javase/7/docs/api/java/lang/Enum.html?is-external=true#clone--;
 title="class or interface in java.lang">clone, 

[13/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/ForwardingPTransform.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/ForwardingPTransform.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/ForwardingPTransform.html
new file mode 100644
index 000..bf6b7f5
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/ForwardingPTransform.html
@@ -0,0 +1,462 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+ForwardingPTransform (Apache Beam SDK for Java, version 
0.2.0-incubating)
+
+
+
+
+
+
+var methods = {"i0":10,"i1":6,"i2":10,"i3":10,"i4":10,"i5":10};
+var tabs = {65535:["t0","All Methods"],2:["t2","Instance 
Methods"],4:["t3","Abstract Methods"],8:["t4","Concrete Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.direct
+Class 
ForwardingPTransformInputT extends PInput,OutputT extends POutput
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">java.lang.Object
+
+
+org.apache.beam.sdk.transforms.PTransformInputT,OutputT
+
+
+org.apache.beam.runners.direct.ForwardingPTransformInputT,OutputT
+
+
+
+
+
+
+
+
+
+All Implemented Interfaces:
+http://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html?is-external=true;
 title="class or interface in java.io">Serializable, HasDisplayData
+
+
+
+public abstract class ForwardingPTransformInputT extends PInput,OutputT extends POutput
+extends PTransformInputT,OutputT
+A base class for implementing PTransform 
overrides, which behave identically to the
+ delegate transform but with overridden methods. Implementors are required to 
implement
+ delegate(),
 which returns the object to forward calls to, and apply(PInput).
+
+See Also:
+Serialized
 Form
+
+
+
+
+
+
+
+
+
+
+
+
+Field Summary
+
+
+
+
+Fields inherited from classorg.apache.beam.sdk.transforms.PTransform
+name
+
+
+
+
+
+
+
+
+Constructor Summary
+
+Constructors
+
+Constructor and Description
+
+
+ForwardingPTransform()
+
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsInstance MethodsAbstract MethodsConcrete Methods
+
+Modifier and Type
+Method and Description
+
+
+OutputT
+apply(InputTinput)
+Applies this PTransform on the given 
InputT, and returns its
+ Output.
+
+
+
+protected abstract PTransformInputT,OutputT
+delegate()
+
+
+TCoderT
+getDefaultOutputCoder(InputTinput,
+ TypedPValueToutput)
+Returns the default Coder to use for the given 
output of
+ this single-output PTransform when applied to the given 
input.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">String
+getName()
+Returns the transform name.
+
+
+
+void
+populateDisplayData(DisplayData.Builderbuilder)
+Register display data for the given transform or 
component.
+
+
+
+void
+validate(InputTinput)
+Called before invoking apply (which may be intercepted by 
the runner) to
+ verify this transform is fully specified and applicable to the specified
+ input.
+
+
+
+
+
+
+
+Methods inherited from classorg.apache.beam.sdk.transforms.PTransform
+getDefaultOutputCoder,
 getDefaultOutputCoder,
 getKindString,
 toString
+
+
+
+
+
+Methods inherited from classjava.lang.http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">Object
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#clone--;
 title="class or interface in java.lang">clone, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#equals-java.lang.Object-;
 title="class or interface in java.lang">equals, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#finalize--;
 title="class or interface in java.lang">finalize, 

[30/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptions.Level.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptions.Level.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptions.Level.html
new file mode 100644
index 000..a91b355
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptions.Level.html
@@ -0,0 +1,414 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+DataflowWorkerLoggingOptions.Level (Apache Beam SDK for Java, version 
0.2.0-incubating)
+
+
+
+
+
+
+var methods = {"i0":9,"i1":9};
+var tabs = {65535:["t0","All Methods"],1:["t1","Static 
Methods"],8:["t4","Concrete Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Enum Constants|
+Field|
+Method
+
+
+Detail:
+Enum Constants|
+Field|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.dataflow.options
+Enum 
DataflowWorkerLoggingOptions.Level
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">java.lang.Object
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Enum.html?is-external=true;
 title="class or interface in java.lang">java.lang.EnumDataflowWorkerLoggingOptions.Level
+
+
+org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions.Level
+
+
+
+
+
+
+
+
+
+All Implemented Interfaces:
+http://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html?is-external=true;
 title="class or interface in java.io">Serializable, http://docs.oracle.com/javase/7/docs/api/java/lang/Comparable.html?is-external=true;
 title="class or interface in java.lang">ComparableDataflowWorkerLoggingOptions.Level
+
+
+Enclosing interface:
+DataflowWorkerLoggingOptions
+
+
+
+public static enum DataflowWorkerLoggingOptions.Level
+extends http://docs.oracle.com/javase/7/docs/api/java/lang/Enum.html?is-external=true;
 title="class or interface in java.lang">EnumDataflowWorkerLoggingOptions.Level
+The set of log levels that can be used on the Dataflow 
worker.
+
+
+
+
+
+
+
+
+
+
+
+Enum Constant Summary
+
+Enum Constants
+
+Enum Constant and Description
+
+
+DEBUG
+Level for logging diagnostic messages.
+
+
+
+ERROR
+Level for logging error messages.
+
+
+
+INFO
+Level for logging informational messages.
+
+
+
+OFF
+Special level used to turn off logging.
+
+
+
+TRACE
+Level for logging tracing messages.
+
+
+
+WARN
+Level for logging warning messages.
+
+
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsStatic MethodsConcrete Methods
+
+Modifier and Type
+Method and Description
+
+
+static DataflowWorkerLoggingOptions.Level
+valueOf(http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">Stringname)
+Returns the enum constant of this type with the specified 
name.
+
+
+
+static DataflowWorkerLoggingOptions.Level[]
+values()
+Returns an array containing the constants of this enum 
type, in
+the order they are declared.
+
+
+
+
+
+
+
+Methods inherited from classjava.lang.http://docs.oracle.com/javase/7/docs/api/java/lang/Enum.html?is-external=true;
 title="class or interface in java.lang">Enum
+http://docs.oracle.com/javase/7/docs/api/java/lang/Enum.html?is-external=true#clone--;
 title="class or interface in java.lang">clone, http://docs.oracle.com/javase/7/docs/api/java/lang/Enum.html?is-external=true#compareTo-E-;
 title="class or interface in java.lang">compareTo, http://docs.oracle.com/javase/7/docs/api/java/lang/Enum.html?is-external=true#equals-java.lang.Object-;
 title="class or interface in java.lang">equals, http://docs.oracle.com/javase/7/docs/api/java/lang/Enum.html?is-external=true#finalize--;
 title="class or interface in java.lang">finalize, http://docs.oracle.com/javase/7/docs/api/java/lang/Enum.html?is-external=true#getDeclaringClass--;
 title="class or interface in 

[09/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/Clock.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/Clock.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/Clock.html
new file mode 100644
index 000..7209c59
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/Clock.html
@@ -0,0 +1,217 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Interface org.apache.beam.runners.direct.Clock (Apache Beam SDK 
for Java, version 0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+
+Uses of Interfaceorg.apache.beam.runners.direct.Clock
+
+
+
+
+
+Packages that use Clock
+
+Package
+Description
+
+
+
+org.apache.beam.runners.direct
+
+Defines the PipelineOptions.DirectRunner
+ which executes both Bounded and Unbounded Pipelines on the local 
machine.
+
+
+
+
+
+
+
+
+
+
+Uses of Clock in org.apache.beam.runners.direct
+
+Classes in org.apache.beam.runners.direct
 that implement Clock
+
+Modifier and Type
+Class and Description
+
+
+
+class
+NanosOffsetClock
+A Clock that 
uses http://docs.oracle.com/javase/7/docs/api/java/lang/System.html?is-external=true#nanoTime--;
 title="class or interface in java.lang">System.nanoTime() to 
track the progress of time.
+
+
+
+
+
+Methods in org.apache.beam.runners.direct
 that return Clock
+
+Modifier and Type
+Method and Description
+
+
+
+Clock
+NanosOffsetClock.Factory.create(PipelineOptionsoptions)
+
+
+Clock
+DirectOptions.getClock()
+Gets the Clock used 
by this pipeline.
+
+
+
+
+
+Methods in org.apache.beam.runners.direct
 with parameters of type Clock
+
+Modifier and Type
+Method and Description
+
+
+
+static WatermarkManager
+WatermarkManager.create(Clockclock,
+  http://docs.oracle.com/javase/7/docs/api/java/util/Collection.html?is-external=true;
 title="class or interface in java.util">CollectionAppliedPTransform?,?,?rootTransforms,
+  http://docs.oracle.com/javase/7/docs/api/java/util/Map.html?is-external=true;
 title="class or interface in java.util">MapPValue,http://docs.oracle.com/javase/7/docs/api/java/util/Collection.html?is-external=true;
 title="class or interface in java.util">CollectionAppliedPTransform?,?,?consumers)
+Creates a new WatermarkManager.
+
+
+
+static DirectTimerInternals
+DirectTimerInternals.create(Clockclock,
+  WatermarkManager.TransformWatermarkswatermarks,
+  WatermarkManager.TimerUpdate.TimerUpdateBuildertimerUpdateBuilder)
+
+
+void
+DirectOptions.setClock(Clockclock)
+
+
+
+
+
+
+
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+Copyright  2016 http://www.apache.org/;>The Apache Software Foundation. All rights 
reserved.
+
+

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/ConsumerTrackingPipelineVisitor.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/ConsumerTrackingPipelineVisitor.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/ConsumerTrackingPipelineVisitor.html
new file mode 100644
index 000..93e0afe
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/ConsumerTrackingPipelineVisitor.html
@@ -0,0 +1,126 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Class 
org.apache.beam.runners.direct.ConsumerTrackingPipelineVisitor (Apache Beam SDK 
for Java, version 0.2.0-incubating)
+
+
+
+
+
+
+var methods = {"i0":10};
+var tabs = {65535:["t0","All Methods"],2:["t2","Instance 
Methods"],8:["t4","Concrete Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.dataflow.util
+Class 
MonitoringUtil.LoggingHandler
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">java.lang.Object
+
+
+org.apache.beam.runners.dataflow.util.MonitoringUtil.LoggingHandler
+
+
+
+
+
+
+
+All Implemented Interfaces:
+MonitoringUtil.JobMessagesHandler
+
+
+Enclosing class:
+MonitoringUtil
+
+
+
+public static class MonitoringUtil.LoggingHandler
+extends http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">Object
+implements MonitoringUtil.JobMessagesHandler
+A handler that logs monitoring messages.
+
+
+
+
+
+
+
+
+
+
+
+Constructor Summary
+
+Constructors
+
+Constructor and Description
+
+
+LoggingHandler()
+
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsInstance MethodsConcrete Methods
+
+Modifier and Type
+Method and Description
+
+
+void
+process(http://docs.oracle.com/javase/7/docs/api/java/util/List.html?is-external=true;
 title="class or interface in 
java.util">Listcom.google.api.services.dataflow.model.JobMessagemessages)
+Process the rows.
+
+
+
+
+
+
+
+Methods inherited from classjava.lang.http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">Object
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#clone--;
 title="class or interface in java.lang">clone, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#equals-java.lang.Object-;
 title="class or interface in java.lang">equals, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#finalize--;
 title="class or interface in java.lang">finalize, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#getClass--;
 title="class or interface in java.lang">getClass, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#hashCode--;
 title="class or interface in java.lang">hashCode, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#notify--;
 title="class or interface in java.lang">notify, http://docs.oracle.com/javase/7/docs/api/java/lang
 /Object.html?is-external=true#notifyAll--" title="class or interface in 
java.lang">notifyAll, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#toString--;
 title="class or interface in java.lang">toString, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#wait--;
 title="class or interface in java.lang">wait, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#wait-long-;
 title="class or interface in java.lang">wait, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#wait-long-int-;
 title="class or interface in java.lang">wait
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Constructor Detail
+
+
+
+
+
+LoggingHandler
+publicLoggingHandler()
+
+
+
+
+
+
+
+
+
+Method Detail
+
+
+
+
+
+process

[25/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/package-use.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/package-use.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/package-use.html
new file mode 100644
index 000..7257da6
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/package-use.html
@@ -0,0 +1,318 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Package org.apache.beam.runners.dataflow.options (Apache Beam 
SDK for Java, version 0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+
+Uses of Packageorg.apache.beam.runners.dataflow.options
+
+
+
+
+
+Packages that use org.apache.beam.runners.dataflow.options
+
+Package
+Description
+
+
+
+org.apache.beam.runners.dataflow
+
+Provides a Beam runner that executes pipelines on the 
Google Cloud Dataflow service.
+
+
+
+org.apache.beam.runners.dataflow.options
+
+Provides PipelineOptions specific to 
Google Cloud Dataflow.
+
+
+
+org.apache.beam.runners.dataflow.testing
+
+Provides utilities for integration testing and RunnableOnService tests of the 
Google Cloud Dataflow
+ runner.
+
+
+
+org.apache.beam.runners.dataflow.util
+
+Provides miscellaneous internal utilities used by the 
Google Cloud Dataflow runner.
+
+
+
+
+
+
+
+
+
+Classes in org.apache.beam.runners.dataflow.options
 used by org.apache.beam.runners.dataflow
+
+Class and Description
+
+
+
+BlockingDataflowPipelineOptions
+Options that are used to configure the BlockingDataflowRunner.
+
+
+
+DataflowPipelineOptions
+Options that can be used to configure the DataflowRunner.
+
+
+
+
+
+
+
+
+
+Classes in org.apache.beam.runners.dataflow.options
 used by org.apache.beam.runners.dataflow.options
+
+Class and Description
+
+
+
+CloudDebuggerOptions
+Options for controlling Cloud Debugger.
+
+
+
+DataflowPipelineDebugOptions
+Internal.
+
+
+
+DataflowPipelineOptions
+Options that can be used to configure the DataflowRunner.
+
+
+
+DataflowPipelineWorkerPoolOptions
+Options that are used to configure the Dataflow pipeline 
worker pool.
+
+
+
+DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType
+Type of autoscaling algorithm to use.
+
+
+
+DataflowPipelineWorkerPoolOptions.TeardownPolicy
+The policy for tearing down the workers spun up by the 
service.
+
+
+
+DataflowProfilingOptions
+Options for controlling profiling of pipeline 
execution.
+
+
+
+DataflowProfilingOptions.DataflowProfilingAgentConfiguration
+Configuration the for profiling agent.
+
+
+
+DataflowWorkerLoggingOptions
+Options that are used to control logging configuration on 
the Dataflow worker.
+
+
+
+DataflowWorkerLoggingOptions.Level
+The set of log levels that can be used on the Dataflow 
worker.
+
+
+
+DataflowWorkerLoggingOptions.WorkerLogLevelOverrides
+Defines a log level override for a specific class, package, 
or name.
+
+
+
+
+
+
+
+
+
+Classes in org.apache.beam.runners.dataflow.options
 used by org.apache.beam.runners.dataflow.testing
+
+Class and Description
+
+
+
+BlockingDataflowPipelineOptions
+Options that are used to configure the BlockingDataflowRunner.
+
+
+
+CloudDebuggerOptions
+Options for controlling Cloud Debugger.
+
+
+
+DataflowPipelineDebugOptions
+Internal.
+
+
+
+DataflowPipelineOptions
+Options that can be used to configure the DataflowRunner.
+
+
+
+DataflowPipelineWorkerPoolOptions
+Options that are used to configure the Dataflow pipeline 
worker pool.
+
+
+
+DataflowProfilingOptions
+Options for controlling profiling of pipeline 
execution.
+
+
+
+DataflowWorkerLoggingOptions
+Options that are used to control logging configuration on 
the Dataflow worker.
+
+
+
+
+
+
+
+
+
+Classes in org.apache.beam.runners.dataflow.options
 used by org.apache.beam.runners.dataflow.util
+
+Class and Description
+
+
+
+DataflowPipelineOptions
+Options that can be used to configure the DataflowRunner.
+
+
+
+
+
+
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+

[04/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/flink/FlinkPipelineOptions.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/flink/FlinkPipelineOptions.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/flink/FlinkPipelineOptions.html
new file mode 100644
index 000..ad83f18
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/flink/FlinkPipelineOptions.html
@@ -0,0 +1,502 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+FlinkPipelineOptions (Apache Beam SDK for Java, version 
0.2.0-incubating)
+
+
+
+
+
+
+var methods = 
{"i0":6,"i1":6,"i2":6,"i3":6,"i4":6,"i5":6,"i6":6,"i7":6,"i8":6,"i9":6,"i10":6,"i11":6,"i12":6,"i13":6};
+var tabs = {65535:["t0","All Methods"],2:["t2","Instance 
Methods"],4:["t3","Abstract Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.flink
+Interface 
FlinkPipelineOptions
+
+
+
+
+
+
+All Superinterfaces:
+ApplicationNameOptions, GcpOptions, GoogleApiDebugOptions, HasDisplayData, PipelineOptions, StreamingOptions
+
+
+
+public interface FlinkPipelineOptions
+extends PipelineOptions, ApplicationNameOptions, StreamingOptions
+Options which can be used to configure a Flink 
PipelineRunner.
+
+
+
+
+
+
+
+
+
+
+
+Nested Class Summary
+
+Nested Classes
+
+Modifier and Type
+Interface and Description
+
+
+static class
+FlinkPipelineOptions.JobNameFactory
+
+
+
+
+
+
+Nested classes/interfaces inherited from 
interfaceorg.apache.beam.sdk.options.GcpOptions
+GcpOptions.CredentialDirFactory, GcpOptions.DefaultProjectFactory, GcpOptions.GcpTempLocationFactory, GcpOptions.GcpUserCredentialsFactory
+
+
+
+
+
+Nested classes/interfaces inherited from 
interfaceorg.apache.beam.sdk.options.GoogleApiDebugOptions
+GoogleApiDebugOptions.GoogleApiTracer
+
+
+
+
+
+Nested classes/interfaces inherited from 
interfaceorg.apache.beam.sdk.options.PipelineOptions
+PipelineOptions.CheckEnabled, 
PipelineOptions.DirectRunner
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsInstance MethodsAbstract Methods
+
+Modifier and Type
+Method and Description
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Long.html?is-external=true;
 title="class or interface in java.lang">Long
+getCheckpointingInterval()
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Long.html?is-external=true;
 title="class or interface in java.lang">Long
+getExecutionRetryDelay()
+
+
+http://docs.oracle.com/javase/7/docs/api/java/util/List.html?is-external=true;
 title="class or interface in java.util">Listhttp://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">String
+getFilesToStage()
+List of local files to make available to workers.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">String
+getFlinkMaster()
+The url of the Flink JobManager on which to execute 
pipelines.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">String
+getJobName()
+The job name is used to identify jobs running on a Flink 
cluster.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html?is-external=true;
 title="class or interface in java.lang">Integer
+getNumberOfExecutionRetries()
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html?is-external=true;
 title="class or interface in java.lang">Integer
+getParallelism()
+
+
+void
+setCheckpointingInterval(http://docs.oracle.com/javase/7/docs/api/java/lang/Long.html?is-external=true;
 title="class or interface in 
java.lang">Longinterval)
+
+
+void
+setExecutionRetryDelay(http://docs.oracle.com/javase/7/docs/api/java/lang/Long.html?is-external=true;
 title="class or interface in java.lang">Longdelay)
+
+
+void
+setFilesToStage(http://docs.oracle.com/javase/7/docs/api/java/util/List.html?is-external=true;
 

[16/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/ConsumerTrackingPipelineVisitor.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/ConsumerTrackingPipelineVisitor.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/ConsumerTrackingPipelineVisitor.html
new file mode 100644
index 000..cbc2b51
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/ConsumerTrackingPipelineVisitor.html
@@ -0,0 +1,482 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+ConsumerTrackingPipelineVisitor (Apache Beam SDK for Java, version 
0.2.0-incubating)
+
+
+
+
+
+
+var methods = 
{"i0":10,"i1":10,"i2":10,"i3":10,"i4":10,"i5":10,"i6":10,"i7":10,"i8":10};
+var tabs = {65535:["t0","All Methods"],2:["t2","Instance 
Methods"],8:["t4","Concrete Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.direct
+Class 
ConsumerTrackingPipelineVisitor
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">java.lang.Object
+
+
+org.apache.beam.sdk.Pipeline.PipelineVisitor.Defaults
+
+
+org.apache.beam.runners.direct.ConsumerTrackingPipelineVisitor
+
+
+
+
+
+
+
+
+
+All Implemented Interfaces:
+Pipeline.PipelineVisitor
+
+
+
+public class ConsumerTrackingPipelineVisitor
+extends Pipeline.PipelineVisitor.Defaults
+Tracks the AppliedPTransforms that 
consume each PValue in the
+ Pipeline. This is used to schedule 
consuming PTransforms to 
consume
+ input after the upstream transform has produced and committed output.
+
+
+
+
+
+
+
+
+
+
+
+Nested Class Summary
+
+
+
+
+Nested classes/interfaces inherited from 
interfaceorg.apache.beam.sdk.Pipeline.PipelineVisitor
+Pipeline.PipelineVisitor.CompositeBehavior, Pipeline.PipelineVisitor.Defaults
+
+
+
+
+
+
+
+
+Constructor Summary
+
+Constructors
+
+Constructor and Description
+
+
+ConsumerTrackingPipelineVisitor()
+
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsInstance MethodsConcrete Methods
+
+Modifier and Type
+Method and Description
+
+
+Pipeline.PipelineVisitor.CompositeBehavior
+enterCompositeTransform(TransformTreeNodenode)
+Called for each composite transform after all topological 
predecessors have been visited
+ but before any of its component transforms.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/util/Collection.html?is-external=true;
 title="class or interface in java.util">CollectionAppliedPTransform?,?,?
+getRootTransforms()
+Returns the root transforms of the Pipeline.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/util/Map.html?is-external=true;
 title="class or interface in java.util">MapAppliedPTransform?,?,?,http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">String
+getStepNames()
+Returns the mapping for each AppliedPTransform in the Pipeline to a unique step
+ name.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/util/Set.html?is-external=true;
 title="class or interface in java.util">SetPValue
+getUnfinalizedPValues()
+Returns all of the PValues that have been produced 
but not consumed.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/util/Map.html?is-external=true;
 title="class or interface in java.util">MapPValue,http://docs.oracle.com/javase/7/docs/api/java/util/Collection.html?is-external=true;
 title="class or interface in java.util">CollectionAppliedPTransform?,?,?
+getValueToConsumers()
+Returns a mapping of each fully-expanded PValue to each
+ AppliedPTransform that 
consumes it.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/util/Collection.html?is-external=true;
 title="class or interface in java.util">CollectionPCollectionView?
+getViews()
+Returns all of the PCollectionViews contained in the 
visited
+ Pipeline.
+
+
+
+void
+leaveCompositeTransform(TransformTreeNodenode)
+Called for each composite transform after 

[17/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/Stager.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/Stager.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/Stager.html
new file mode 100644
index 000..94137bc
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/Stager.html
@@ -0,0 +1,242 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Interface org.apache.beam.runners.dataflow.util.Stager (Apache 
Beam SDK for Java, version 0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+
+Uses of 
Interfaceorg.apache.beam.runners.dataflow.util.Stager
+
+
+
+
+
+Packages that use Stager
+
+Package
+Description
+
+
+
+org.apache.beam.runners.dataflow.options
+
+Provides PipelineOptions specific to 
Google Cloud Dataflow.
+
+
+
+org.apache.beam.runners.dataflow.util
+
+Provides miscellaneous internal utilities used by the 
Google Cloud Dataflow runner.
+
+
+
+
+
+
+
+
+
+
+Uses of Stager in org.apache.beam.runners.dataflow.options
+
+Methods in org.apache.beam.runners.dataflow.options
 that return Stager
+
+Modifier and Type
+Method and Description
+
+
+
+Stager
+DataflowPipelineDebugOptions.StagerFactory.create(PipelineOptionsoptions)
+
+
+Stager
+DataflowPipelineDebugOptions.getStager()
+The resource stager instance that should be used to stage 
resources.
+
+
+
+
+
+Methods in org.apache.beam.runners.dataflow.options
 that return types with arguments of type Stager
+
+Modifier and Type
+Method and Description
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Class.html?is-external=true;
 title="class or interface in java.lang">Class? extends Stager
+DataflowPipelineDebugOptions.getStagerClass()
+The class responsible for staging resources to be 
accessible by workers
+ during job execution.
+
+
+
+
+
+Methods in org.apache.beam.runners.dataflow.options
 with parameters of type Stager
+
+Modifier and Type
+Method and Description
+
+
+
+void
+DataflowPipelineDebugOptions.setStager(Stagerstager)
+
+
+
+
+Method parameters in org.apache.beam.runners.dataflow.options
 with type arguments of type Stager
+
+Modifier and Type
+Method and Description
+
+
+
+void
+DataflowPipelineDebugOptions.setStagerClass(http://docs.oracle.com/javase/7/docs/api/java/lang/Class.html?is-external=true;
 title="class or interface in java.lang">Class? extends StagerstagerClass)
+
+
+
+
+
+
+
+Uses of Stager in org.apache.beam.runners.dataflow.util
+
+Classes in org.apache.beam.runners.dataflow.util
 that implement Stager
+
+Modifier and Type
+Class and Description
+
+
+
+class
+GcsStager
+Utility class for staging files to GCS.
+
+
+
+
+
+
+
+
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+Copyright  2016 http://www.apache.org/;>The Apache Software Foundation. All rights 
reserved.
+
+

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/TimeUtil.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/TimeUtil.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/TimeUtil.html
new file mode 100644
index 000..8ab446c
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/util/class-use/TimeUtil.html
@@ -0,0 +1,126 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Class org.apache.beam.runners.dataflow.util.TimeUtil (Apache 
Beam SDK for Java, version 0.2.0-incubating)
+
+
+
+
+
+
+var methods = 
{"i0":6,"i1":6,"i2":6,"i3":6,"i4":6,"i5":6,"i6":6,"i7":6,"i8":6,"i9":6,"i10":6,"i11":6,"i12":6,"i13":6,"i14":6,"i15":6,"i16":6,"i17":6,"i18":6,"i19":6,"i20":6,"i21":6,"i22":6,"i23":6,"i24":6,"i25":6,"i26":6,"i27":6,"i28":6,"i29":6};
+var tabs = {65535:["t0","All Methods"],2:["t2","Instance 
Methods"],4:["t3","Abstract Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.dataflow.options
+Interface 
DataflowPipelineDebugOptions
+
+
+
+
+
+
+All Superinterfaces:
+HasDisplayData, PipelineOptions
+
+
+All Known Subinterfaces:
+BlockingDataflowPipelineOptions, 
DataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions
+
+
+
+@Hidden
+public interface DataflowPipelineDebugOptions
+extends PipelineOptions
+Internal. Options used to control execution of the Dataflow 
SDK for
+ debugging and testing purposes.
+
+
+
+
+
+
+
+
+
+
+
+Nested Class Summary
+
+Nested Classes
+
+Modifier and Type
+Interface and Description
+
+
+static class
+DataflowPipelineDebugOptions.DataflowClientFactory
+Returns the default Dataflow client built from the passed 
in PipelineOptions.
+
+
+
+static class
+DataflowPipelineDebugOptions.PathValidatorFactory
+Creates a PathValidator 
object using the class specified in
+ getPathValidatorClass().
+
+
+
+static class
+DataflowPipelineDebugOptions.StagerFactory
+Creates a Stager object using the 
class specified in
+ getStagerClass().
+
+
+
+
+
+
+
+Nested classes/interfaces inherited from 
interfaceorg.apache.beam.sdk.options.PipelineOptions
+PipelineOptions.CheckEnabled, 
PipelineOptions.DirectRunner
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsInstance MethodsAbstract Methods
+
+Modifier and Type
+Method and Description
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">String
+getApiRootUrl()
+The root URL for the Dataflow API.
+
+
+
+com.google.api.services.dataflow.Dataflow
+getDataflowClient()
+An instance of the Dataflow client.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">String
+getDataflowEndpoint()
+Dataflow endpoint to use.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">String
+getDataflowJobFile()
+The path to write the translated Dataflow job specification 
out to
+ at job submission time.
+
+
+
+boolean
+getDumpHeapOnOOM()
+If true, save a heap dump before killing a thread or 
process which is GC
+ thrashing or out of memory.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/util/List.html?is-external=true;
 title="class or interface in java.util">Listhttp://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">String
+getExperiments()
+The list of backend experiments to enable.
+
+
+
+int
+getNumberOfWorkerHarnessThreads()
+Number of threads to use on the Dataflow worker 
harness.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">String
+getOverrideWindmillBinary()
+Custom windmill_main binary to use with the streaming 

[06/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/WatermarkManager.TimerUpdate.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/WatermarkManager.TimerUpdate.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/WatermarkManager.TimerUpdate.html
new file mode 100644
index 000..41a07ff
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/WatermarkManager.TimerUpdate.html
@@ -0,0 +1,220 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Class 
org.apache.beam.runners.direct.WatermarkManager.TimerUpdate (Apache Beam SDK 
for Java, version 0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+
+Uses 
of Classorg.apache.beam.runners.direct.WatermarkManager.TimerUpdate
+
+
+
+
+
+Packages that use WatermarkManager.TimerUpdate
+
+Package
+Description
+
+
+
+org.apache.beam.runners.direct
+
+Defines the PipelineOptions.DirectRunner
+ which executes both Bounded and Unbounded Pipelines on the local 
machine.
+
+
+
+
+
+
+
+
+
+
+Uses of WatermarkManager.TimerUpdate in org.apache.beam.runners.direct
+
+Methods in org.apache.beam.runners.direct
 that return WatermarkManager.TimerUpdate
+
+Modifier and Type
+Method and Description
+
+
+
+WatermarkManager.TimerUpdate
+WatermarkManager.TimerUpdate.TimerUpdateBuilder.build()
+Returns a new WatermarkManager.TimerUpdate 
with the most recently set completedTimers, setTimers,
+ and deletedTimers.
+
+
+
+static WatermarkManager.TimerUpdate
+WatermarkManager.TimerUpdate.empty()
+Returns a TimerUpdate for a null key with no timers.
+
+
+
+WatermarkManager.TimerUpdate
+TransformResult.getTimerUpdate()
+Returns a TimerUpdateBuilder that was produced as a result 
of this evaluation.
+
+
+
+WatermarkManager.TimerUpdate
+StepTransformResult.getTimerUpdate()
+
+
+WatermarkManager.TimerUpdate
+DirectTimerInternals.getTimerUpdate()
+
+
+WatermarkManager.TimerUpdate
+WatermarkManager.TimerUpdate.withCompletedTimers(http://docs.oracle.com/javase/7/docs/api/java/lang/Iterable.html?is-external=true;
 title="class or interface in java.lang">IterableTimerInternals.TimerDatacompletedTimers)
+Returns a WatermarkManager.TimerUpdate 
that is like this one, but with the specified completed timers.
+
+
+
+
+
+Methods in org.apache.beam.runners.direct
 with parameters of type WatermarkManager.TimerUpdate
+
+Modifier and Type
+Method and Description
+
+
+
+void
+WatermarkManager.updateWatermarks(DirectRunner.CommittedBundle?completed,
+WatermarkManager.TimerUpdatetimerUpdate,
+org.apache.beam.runners.direct.CommittedResultresult,
+org.joda.time.InstantearliestHold)
+Updates the watermarks of a transform with one or more 
inputs.
+
+
+
+StepTransformResult.Builder
+StepTransformResult.Builder.withTimerUpdate(WatermarkManager.TimerUpdatetimerUpdate)
+
+
+
+
+
+
+
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+Copyright  2016 http://www.apache.org/;>The Apache Software Foundation. All rights 
reserved.
+
+

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/WatermarkManager.TransformWatermarks.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/WatermarkManager.TransformWatermarks.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/WatermarkManager.TransformWatermarks.html
new file mode 100644
index 000..c35ac21
--- /dev/null
+++ 

[46/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/deprecated-list.html
--
diff --git a/content/learn/sdks/javadoc/0.2.0-incubating/deprecated-list.html 
b/content/learn/sdks/javadoc/0.2.0-incubating/deprecated-list.html
new file mode 100644
index 000..a0fedd5
--- /dev/null
+++ b/content/learn/sdks/javadoc/0.2.0-incubating/deprecated-list.html
@@ -0,0 +1,368 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Deprecated List (Apache Beam SDK for Java, version 
0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+
+Deprecated API
+Contents
+
+Deprecated Classes
+Deprecated Fields
+Deprecated Methods
+Deprecated Constructors
+Deprecated Enum Constants
+
+
+
+
+
+
+
+
+Deprecated Classes
+
+Class and Description
+
+
+
+org.apache.beam.sdk.transforms.Combine.SimpleCombineFn
+
+
+
+
+
+
+
+
+
+
+
+Deprecated Fields
+
+Field and Description
+
+
+
+org.apache.beam.sdk.util.AppEngineEnvironment.IS_APP_ENGINE
+
+
+
+
+
+
+
+
+
+
+
+Deprecated Methods
+
+Method and Description
+
+
+
+org.apache.beam.sdk.io.PubsubIO.PubsubSubscription.asV1Beta1Path()
+the v1beta1 API for Cloud 
Pub/Sub is deprecated.
+
+
+
+org.apache.beam.sdk.io.PubsubIO.PubsubTopic.asV1Beta1Path()
+the v1beta1 API for Cloud 
Pub/Sub is deprecated.
+
+
+
+org.apache.beam.sdk.io.PubsubIO.PubsubSubscription.asV1Beta2Path()
+the v1beta2 API for Cloud 
Pub/Sub is deprecated.
+
+
+
+org.apache.beam.sdk.io.PubsubIO.PubsubTopic.asV1Beta2Path()
+the v1beta2 API for Cloud 
Pub/Sub is deprecated.
+
+
+
+org.apache.beam.sdk.util.common.Counter.booleans(String,
 Counter.AggregationKind)
+use Counter.booleans(CounterName,
 AggregationKind).
+
+
+
+org.apache.beam.sdk.coders.AvroCoder.createDatumReader()
+For AvroCoder 
internal use only.
+
+
+
+org.apache.beam.sdk.coders.AvroCoder.createDatumWriter()
+For AvroCoder 
internal use only.
+
+
+
+org.apache.beam.runners.dataflow.util.PackageUtil.createPackage(File,
 String, String)
+
+
+org.apache.beam.sdk.util.common.Counter.doubles(String,
 Counter.AggregationKind)
+use Counter.doubles(CounterName,
 AggregationKind).
+
+
+
+org.apache.beam.sdk.transforms.DoFn.getAllowedTimestampSkew()
+does not interact well 
with the watermark.
+
+
+
+org.apache.beam.sdk.Pipeline.getFullNameForTesting(PTransform?,
 ?)
+this method is no longer 
compatible with the design of Pipeline,
+ as PTransforms can be applied 
multiple times, with different names
+ each time.
+
+
+
+org.apache.beam.sdk.util.common.Counter.getName()
+use Counter.getFlatName().
+
+
+
+org.apache.beam.sdk.util.common.Counter.ints(String,
 Counter.AggregationKind)
+use Counter.ints(CounterName,
 AggregationKind).
+
+
+
+org.apache.beam.sdk.util.common.Counter.longs(String,
 Counter.AggregationKind)
+use Counter.longs(CounterName,
 AggregationKind).
+
+
+
+org.apache.beam.sdk.transforms.windowing.AfterDelayFromFirstElement.mappedTo(SerializableFunctionInstant,
 Instant)
+This will be removed in 
the next major version. Please use only
+ AfterDelayFromFirstElement.plusDelayOf(org.joda.time.Duration)
 and AfterDelayFromFirstElement.alignedTo(org.joda.time.Duration,
 org.joda.time.Instant).
+
+
+
+org.apache.beam.sdk.util.Transport.newPubsubClient(PubsubOptions)
+Use an appropriate
+ PubsubClient.PubsubClientFactory
+
+
+
+org.apache.beam.sdk.transforms.Combine.SimpleCombineFn.of(SerializableFunctionIterableV,
 V)
+
+
+org.apache.beam.sdk.coders.protobuf.ProtoCoder.of(String,
 ListString)
+For JSON deserialization 
only.
+
+
+
+org.apache.beam.sdk.io.CountingSource.unbounded()
+use CountingInput.unbounded()
 instead
+
+
+
+org.apache.beam.sdk.io.CountingSource.unboundedWithTimestampFn(SerializableFunctionLong,
 Instant)
+use CountingInput.unbounded()
 and call
+ CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction)
 instead
+
+
+
+org.apache.beam.sdk.io.CountingSource.upTo(long)
+use CountingInput.upTo(long)
 instead
+
+
+
+org.apache.beam.sdk.util.WindowedValue.valueInEmptyWindows(T)
+a value in no windows 
technically is not "in" a PCollection. It is allowed to drop
+ it at any point, and benign runner implementation details could cause 
silent data loss.
+
+
+
+org.apache.beam.sdk.util.WindowedValue.valueInEmptyWindows(T,
 PaneInfo)
+a 

[27/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/class-use/DataflowWorkerLoggingOptions.Level.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/class-use/DataflowWorkerLoggingOptions.Level.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/class-use/DataflowWorkerLoggingOptions.Level.html
new file mode 100644
index 000..11c72a7
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/options/class-use/DataflowWorkerLoggingOptions.Level.html
@@ -0,0 +1,237 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Class 
org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions.Level 
(Apache Beam SDK for Java, version 0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+
+Uses of 
Classorg.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions.Level
+
+
+
+
+
+Packages that use DataflowWorkerLoggingOptions.Level
+
+Package
+Description
+
+
+
+org.apache.beam.runners.dataflow.options
+
+Provides PipelineOptions specific to 
Google Cloud Dataflow.
+
+
+
+
+
+
+
+
+
+
+Uses of DataflowWorkerLoggingOptions.Level
 in org.apache.beam.runners.dataflow.options
+
+Methods in org.apache.beam.runners.dataflow.options
 that return DataflowWorkerLoggingOptions.Level
+
+Modifier and Type
+Method and Description
+
+
+
+DataflowWorkerLoggingOptions.Level
+DataflowWorkerLoggingOptions.getDefaultWorkerLogLevel()
+This option controls the default log level of all loggers 
without a log level override.
+
+
+
+DataflowWorkerLoggingOptions.Level
+DataflowWorkerLoggingOptions.getWorkerSystemErrMessageLevel()
+Controls the log level given to messages printed to 
System.err.
+
+
+
+DataflowWorkerLoggingOptions.Level
+DataflowWorkerLoggingOptions.getWorkerSystemOutMessageLevel()
+Controls the log level given to messages printed to 
System.out.
+
+
+
+static DataflowWorkerLoggingOptions.Level
+DataflowWorkerLoggingOptions.Level.valueOf(http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">Stringname)
+Returns the enum constant of this type with the specified 
name.
+
+
+
+static DataflowWorkerLoggingOptions.Level[]
+DataflowWorkerLoggingOptions.Level.values()
+Returns an array containing the constants of this enum 
type, in
+the order they are declared.
+
+
+
+
+
+Methods in org.apache.beam.runners.dataflow.options
 with parameters of type DataflowWorkerLoggingOptions.Level
+
+Modifier and Type
+Method and Description
+
+
+
+DataflowWorkerLoggingOptions.WorkerLogLevelOverrides
+DataflowWorkerLoggingOptions.WorkerLogLevelOverrides.addOverrideForClass(http://docs.oracle.com/javase/7/docs/api/java/lang/Class.html?is-external=true;
 title="class or interface in java.lang">Class?klass,
+   DataflowWorkerLoggingOptions.Levellevel)
+Overrides the default log level for the passed in 
class.
+
+
+
+DataflowWorkerLoggingOptions.WorkerLogLevelOverrides
+DataflowWorkerLoggingOptions.WorkerLogLevelOverrides.addOverrideForName(http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">Stringname,
+  DataflowWorkerLoggingOptions.Levellevel)
+Overrides the default log level for the passed in 
name.
+
+
+
+DataflowWorkerLoggingOptions.WorkerLogLevelOverrides
+DataflowWorkerLoggingOptions.WorkerLogLevelOverrides.addOverrideForPackage(http://docs.oracle.com/javase/7/docs/api/java/lang/Package.html?is-external=true;
 title="class or interface in java.lang">Packagepkg,
+ DataflowWorkerLoggingOptions.Levellevel)
+Overrides the default log level for the passed in 
package.
+
+
+
+void
+DataflowWorkerLoggingOptions.setDefaultWorkerLogLevel(DataflowWorkerLoggingOptions.Levellevel)
+
+
+void
+DataflowWorkerLoggingOptions.setWorkerSystemErrMessageLevel(DataflowWorkerLoggingOptions.Levellevel)
+
+
+void
+DataflowWorkerLoggingOptions.setWorkerSystemOutMessageLevel(DataflowWorkerLoggingOptions.Levellevel)
+
+
+
+
+
+
+
+
+
+
+

[07/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/StepTransformResult.Builder.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/StepTransformResult.Builder.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/StepTransformResult.Builder.html
new file mode 100644
index 000..302b769
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/StepTransformResult.Builder.html
@@ -0,0 +1,199 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Class 
org.apache.beam.runners.direct.StepTransformResult.Builder (Apache Beam SDK for 
Java, version 0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+
+Uses 
of Classorg.apache.beam.runners.direct.StepTransformResult.Builder
+
+
+
+
+
+Packages that use StepTransformResult.Builder
+
+Package
+Description
+
+
+
+org.apache.beam.runners.direct
+
+Defines the PipelineOptions.DirectRunner
+ which executes both Bounded and Unbounded Pipelines on the local 
machine.
+
+
+
+
+
+
+
+
+
+
+Uses of StepTransformResult.Builder in org.apache.beam.runners.direct
+
+Methods in org.apache.beam.runners.direct
 that return StepTransformResult.Builder
+
+Modifier and Type
+Method and Description
+
+
+
+StepTransformResult.Builder
+StepTransformResult.Builder.addOutput(http://docs.oracle.com/javase/7/docs/api/java/util/Collection.html?is-external=true;
 title="class or interface in java.util">CollectionDirectRunner.UncommittedBundle?outputBundles)
+
+
+StepTransformResult.Builder
+StepTransformResult.Builder.addOutput(DirectRunner.UncommittedBundle?outputBundle,
+ DirectRunner.UncommittedBundle?...outputBundles)
+
+
+StepTransformResult.Builder
+StepTransformResult.Builder.addUnprocessedElements(http://docs.oracle.com/javase/7/docs/api/java/lang/Iterable.html?is-external=true;
 title="class or interface in java.lang">Iterable? extends WindowedValue?unprocessed)
+
+
+StepTransformResult.Builder
+StepTransformResult.Builder.withCounters(CounterSetcounters)
+
+
+static StepTransformResult.Builder
+StepTransformResult.withHold(AppliedPTransform?,?,?transform,
+org.joda.time.InstantwatermarkHold)
+
+
+static StepTransformResult.Builder
+StepTransformResult.withoutHold(AppliedPTransform?,?,?transform)
+
+
+StepTransformResult.Builder
+StepTransformResult.Builder.withState(CopyOnAccessInMemoryStateInternals?state)
+
+
+StepTransformResult.Builder
+StepTransformResult.Builder.withTimerUpdate(WatermarkManager.TimerUpdatetimerUpdate)
+
+
+
+
+
+
+
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+Copyright  2016 http://www.apache.org/;>The Apache Software Foundation. All rights 
reserved.
+
+

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/StepTransformResult.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/StepTransformResult.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/StepTransformResult.html
new file mode 100644
index 000..242a38f
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/StepTransformResult.html
@@ -0,0 +1,169 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Class org.apache.beam.runners.direct.StepTransformResult 
(Apache Beam SDK for Java, version 0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+
+Hierarchy For Package 
org.apache.beam.runners.dataflow.testing
+Package Hierarchies:
+
+All Packages
+
+
+
+Class Hierarchy
+
+java.lang.http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">Object
+
+org.apache.beam.sdk.runners.PipelineRunnerResultT
+
+org.apache.beam.runners.dataflow.testing.TestDataflowRunner
+
+
+
+
+
+Interface Hierarchy
+
+org.apache.beam.runners.dataflow.options.CloudDebuggerOptions
+
+org.apache.beam.runners.dataflow.options.DataflowPipelineOptions (also extends 
org.apache.beam.sdk.options.ApplicationNameOptions, 
org.apache.beam.sdk.options.BigQueryOptions, 
org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions, 
org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions,
 org.apache.beam.runners.dataflow.options.DataflowProfilingOptions, 
org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions, 
org.apache.beam.sdk.options.GcpOptions, 
org.apache.beam.sdk.options.GcsOptions, 
org.apache.beam.sdk.options.PipelineOptions, 
org.apache.beam.sdk.options.PubsubOptions, 
org.apache.beam.sdk.options.StreamingOptions)
+
+org.apache.beam.runners.dataflow.options.BlockingDataflowPipelineOptions
+
+org.apache.beam.runners.dataflow.testing.TestDataflowPipelineOptions (also extends 
org.apache.beam.sdk.testing.TestPipelineOptions)
+
+
+
+
+
+
+org.apache.beam.runners.dataflow.options.DataflowProfilingOptions
+
+org.apache.beam.runners.dataflow.options.DataflowPipelineOptions (also extends 
org.apache.beam.sdk.options.ApplicationNameOptions, 
org.apache.beam.sdk.options.BigQueryOptions, 
org.apache.beam.runners.dataflow.options.CloudDebuggerOptions, 
org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions, 
org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions,
 org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions, 
org.apache.beam.sdk.options.GcpOptions, 
org.apache.beam.sdk.options.GcsOptions, 
org.apache.beam.sdk.options.PipelineOptions, 
org.apache.beam.sdk.options.PubsubOptions, 
org.apache.beam.sdk.options.StreamingOptions)
+
+org.apache.beam.runners.dataflow.options.BlockingDataflowPipelineOptions
+
+org.apache.beam.runners.dataflow.testing.TestDataflowPipelineOptions (also extends 
org.apache.beam.sdk.testing.TestPipelineOptions)
+
+
+
+
+
+
+org.apache.beam.sdk.transforms.display.HasDisplayData
+
+org.apache.beam.sdk.options.PipelineOptions
+
+org.apache.beam.sdk.options.ApplicationNameOptions
+
+org.apache.beam.sdk.options.BigQueryOptions (also extends 
org.apache.beam.sdk.options.GcpOptions, 
org.apache.beam.sdk.options.PipelineOptions, 
org.apache.beam.sdk.options.StreamingOptions)
+
+org.apache.beam.runners.dataflow.options.DataflowPipelineOptions (also extends 
org.apache.beam.sdk.options.ApplicationNameOptions, 
org.apache.beam.runners.dataflow.options.CloudDebuggerOptions, 
org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions, 
org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions,
 org.apache.beam.runners.dataflow.options.DataflowProfilingOptions, 
org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions, 
org.apache.beam.sdk.options.GcpOptions, 
org.apache.beam.sdk.options.GcsOptions
 , org.apache.beam.sdk.options.PipelineOptions, 
org.apache.beam.sdk.options.PubsubOptions, 
org.apache.beam.sdk.options.StreamingOptions)
+

[08/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/DirectRunner.UncommittedBundle.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/DirectRunner.UncommittedBundle.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/DirectRunner.UncommittedBundle.html
new file mode 100644
index 000..13d67f4
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/direct/class-use/DirectRunner.UncommittedBundle.html
@@ -0,0 +1,258 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Interface 
org.apache.beam.runners.direct.DirectRunner.UncommittedBundle (Apache Beam SDK 
for Java, version 0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+
+Uses of 
Interfaceorg.apache.beam.runners.direct.DirectRunner.UncommittedBundle
+
+
+
+
+
+Packages that use DirectRunner.UncommittedBundle
+
+Package
+Description
+
+
+
+org.apache.beam.runners.direct
+
+Defines the PipelineOptions.DirectRunner
+ which executes both Bounded and Unbounded Pipelines on the local 
machine.
+
+
+
+
+
+
+
+
+
+
+Uses of DirectRunner.UncommittedBundle in org.apache.beam.runners.direct
+
+Methods in org.apache.beam.runners.direct
 that return DirectRunner.UncommittedBundle
+
+Modifier and Type
+Method and Description
+
+
+
+DirectRunner.UncommittedBundleT
+DirectRunner.UncommittedBundle.add(WindowedValueTelement)
+Outputs an element to this bundle.
+
+
+
+TDirectRunner.UncommittedBundleT
+BundleFactory.createBundle(DirectRunner.CommittedBundle?input,
+PCollectionToutput)
+Create an DirectRunner.UncommittedBundle 
from the specified input.
+
+
+
+K,TDirectRunner.UncommittedBundleT
+BundleFactory.createKeyedBundle(DirectRunner.CommittedBundle?input,
+ 
org.apache.beam.runners.direct.StructuralKeyKkey,
+ PCollectionToutput)
+Create an DirectRunner.UncommittedBundle 
with the specified keys at the specified step.
+
+
+
+TDirectRunner.UncommittedBundleT
+BundleFactory.createRootBundle(PCollectionToutput)
+Create an DirectRunner.UncommittedBundle 
from an empty input.
+
+
+
+
+
+Methods in org.apache.beam.runners.direct
 that return types with arguments of type DirectRunner.UncommittedBundle
+
+Modifier and Type
+Method and Description
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Iterable.html?is-external=true;
 title="class or interface in java.lang">Iterable? extends DirectRunner.UncommittedBundle?
+TransformResult.getOutputBundles()
+Returns the (uncommitted) 
Bundles output by this transform.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Iterable.html?is-external=true;
 title="class or interface in java.lang">Iterable? extends DirectRunner.UncommittedBundle?
+StepTransformResult.getOutputBundles()
+
+
+
+
+Methods in org.apache.beam.runners.direct
 with parameters of type DirectRunner.UncommittedBundle
+
+Modifier and Type
+Method and Description
+
+
+
+StepTransformResult.Builder
+StepTransformResult.Builder.addOutput(DirectRunner.UncommittedBundle?outputBundle,
+ DirectRunner.UncommittedBundle?...outputBundles)
+
+
+StepTransformResult.Builder
+StepTransformResult.Builder.addOutput(DirectRunner.UncommittedBundle?outputBundle,
+ DirectRunner.UncommittedBundle?...outputBundles)
+
+
+
+
+Method parameters in org.apache.beam.runners.direct
 with type arguments of type DirectRunner.UncommittedBundle
+
+Modifier and Type
+Method and Description
+
+
+
+StepTransformResult.Builder
+StepTransformResult.Builder.addOutput(http://docs.oracle.com/javase/7/docs/api/java/util/Collection.html?is-external=true;
 title="class or interface in java.util">CollectionDirectRunner.UncommittedBundle?outputBundles)
+
+
+static UncommittedBundleOutputManager
+UncommittedBundleOutputManager.create(http://docs.oracle.com/javase/7/docs/api/java/util/Map.html?is-external=true;
 title="class or interface in java.util">MapTupleTag?,DirectRunner.UncommittedBundle?outputBundles)
+
+
+
+
+Constructor parameters in org.apache.beam.runners.direct
 with type arguments of type 

[41/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/DataflowJobCancelledException.html
--
diff --git 
a/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/DataflowJobCancelledException.html
 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/DataflowJobCancelledException.html
new file mode 100644
index 000..fe28a3b
--- /dev/null
+++ 
b/content/learn/sdks/javadoc/0.2.0-incubating/org/apache/beam/runners/dataflow/DataflowJobCancelledException.html
@@ -0,0 +1,306 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+DataflowJobCancelledException (Apache Beam SDK for Java, version 
0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.dataflow
+Class 
DataflowJobCancelledException
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">java.lang.Object
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html?is-external=true;
 title="class or interface in java.lang">java.lang.Throwable
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Exception.html?is-external=true;
 title="class or interface in java.lang">java.lang.Exception
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/RuntimeException.html?is-external=true;
 title="class or interface in java.lang">java.lang.RuntimeException
+
+
+org.apache.beam.runners.dataflow.DataflowJobException
+
+
+org.apache.beam.runners.dataflow.DataflowJobCancelledException
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+All Implemented Interfaces:
+http://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html?is-external=true;
 title="class or interface in java.io">Serializable
+
+
+
+public class DataflowJobCancelledException
+extends DataflowJobException
+Signals that a job run by a BlockingDataflowRunner was 
updated during execution.
+
+See Also:
+Serialized
 Form
+
+
+
+
+
+
+
+
+
+
+
+
+Constructor Summary
+
+Constructors
+
+Constructor and Description
+
+
+DataflowJobCancelledException(DataflowPipelineJobjob,
+ http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">Stringmessage)
+Create a new 
DataflowJobAlreadyUpdatedException with the specified DataflowPipelineJob and 
message.
+
+
+
+DataflowJobCancelledException(DataflowPipelineJobjob,
+ http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">Stringmessage,
+ http://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html?is-external=true;
 title="class or interface in java.lang">Throwablecause)
+Create a new 
DataflowJobAlreadyUpdatedException with the specified DataflowPipelineJob, 
message, and cause.
+
+
+
+
+
+
+
+
+
+
+Method Summary
+
+
+
+
+Methods inherited from classorg.apache.beam.runners.dataflow.DataflowJobException
+getJob
+
+
+
+
+
+Methods inherited from classjava.lang.http://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html?is-external=true;
 title="class or interface in java.lang">Throwable
+http://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html?is-external=true#addSuppressed-java.lang.Throwable-;
 title="class or interface in java.lang">addSuppressed, http://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html?is-external=true#fillInStackTrace--;
 title="class or interface in java.lang">fillInStackTrace, http://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html?is-external=true#getCause--;
 title="class or interface in java.lang">getCause, http://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html?is-external=true#getLocalizedMessage--;
 title="class or interface in java.lang">getLocalizedMessage, http://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html?is-external=true#getMessage--;
 title="class or interface in java.lang">getMessage, http://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html?is-external=true#getStackTrace--;
 title="class 

[45/52] [abbrv] [partial] incubator-beam-site git commit: Regenerated html

[frances]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/65d23415/content/learn/sdks/javadoc/0.2.0-incubating/index-all.html
--
diff --git a/content/learn/sdks/javadoc/0.2.0-incubating/index-all.html 
b/content/learn/sdks/javadoc/0.2.0-incubating/index-all.html
new file mode 100644
index 000..58d17b1
--- /dev/null
+++ b/content/learn/sdks/javadoc/0.2.0-incubating/index-all.html
@@ -0,0 +1,17910 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Index (Apache Beam SDK for Java, version 0.2.0-incubating)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+ABCDEFGHIJKLMNOPQRSTUVWXYZ
+
+
+A
+
+access(StateTag?
 super K, StateT) - Method in interface 
org.apache.beam.sdk.util.state.StateAccessor
+
+Access the storage for the given address in 
the current window.
+
+accessInEachMergingWindow(StateTag?
 super K, StateT) - Method in interface 
org.apache.beam.sdk.util.state.MergingStateAccessor
+
+Analogous to StateAccessor.access(org.apache.beam.sdk.util.state.StateTag?
 super K, StateT), but returned as a map from each window which 
is
+ about to be merged to the corresponding state.
+
+AccumulatingCombineFn()
 - Constructor for class org.apache.beam.sdk.transforms.Combine.AccumulatingCombineFn
+
+accumulatingFiredPanes()
 - Static method in class org.apache.beam.sdk.transforms.windowing.Window
+
+Returns a new Window PTransform 
that uses the registered WindowFn and
+ Triggering behavior, and that accumulates elements in a pane after they are 
triggered.
+
+accumulatingFiredPanes()
 - Method in class org.apache.beam.sdk.transforms.windowing.Window.Bound
+
+Returns a new Window PTransform 
that uses the registered WindowFn and
+ Triggering behavior, and that accumulates elements in a pane after they are 
triggered.
+
+accumulatingFiredPanes()
 - Method in class org.apache.beam.sdk.transforms.windowing.Window.Unbound
+
+Returns a new Window PTransform 
that uses the registered WindowFn and
+ Triggering behavior, and that accumulates elements in a pane after they are 
triggered.
+
+AccumulatorCombiningStateInputT,AccumT,OutputT - Interface 
in org.apache.beam.sdk.util.state
+
+State for a single value that is managed by a Combine.CombineFn.
+
+ackDeadlineSeconds(PubsubClient.SubscriptionPath)
 - Method in class org.apache.beam.sdk.util.PubsubClient
+
+Return the ack deadline, in seconds, for 
subscription.
+
+ackDeadlineSeconds(PubsubClient.SubscriptionPath)
 - Method in class org.apache.beam.sdk.util.PubsubGrpcClient
+
+ackDeadlineSeconds(PubsubClient.SubscriptionPath)
 - Method in class org.apache.beam.sdk.util.PubsubJsonClient
+
+ackDeadlineSeconds(PubsubClient.SubscriptionPath)
 - Method in class org.apache.beam.sdk.util.PubsubTestClient
+
+ackId
 - Variable in class org.apache.beam.sdk.util.PubsubClient.IncomingMessage
+
+Id to pass back to Pubsub to acknowledge receipt of this 
message.
+
+acknowledge(PubsubClient.SubscriptionPath,
 ListString) - Method in class org.apache.beam.sdk.util.PubsubClient
+
+Acknowldege messages from subscription with 
ackIds.
+
+acknowledge(PubsubClient.SubscriptionPath,
 ListString) - Method in class org.apache.beam.sdk.util.PubsubGrpcClient
+
+acknowledge(PubsubClient.SubscriptionPath,
 ListString) - Method in class org.apache.beam.sdk.util.PubsubJsonClient
+
+acknowledge(PubsubClient.SubscriptionPath,
 ListString) - Method in class org.apache.beam.sdk.util.PubsubTestClient
+
+ActiveWindowSetW extends BoundedWindow - Interface 
in org.apache.beam.sdk.util
+
+Track which windows are active, and the state 
address window(s) under which their
+ state is stored.
+
+ActiveWindowSet.MergeCallbackW extends BoundedWindow - Interface 
in org.apache.beam.sdk.util
+
+Callback for ActiveWindowSet.merge(org.apache.beam.sdk.util.ActiveWindowSet.MergeCallbackW).
+
+add(IterableWindowedValueElemT)
 - Method in interface org.apache.beam.runners.direct.DirectRunner.PCollectionViewWriter
+
+add(WindowedValueT)
 - Method in interface org.apache.beam.runners.direct.DirectRunner.UncommittedBundle
+
+Outputs an element to this bundle.
+
+add(Long)
 - Method in class org.apache.beam.sdk.transforms.ApproximateUnique.ApproximateUniqueCombineFn.LargestUnique
+
+Adds a value to the heap, returning whether the value is 
(large enough
+ to be) in the heap.
+
+add(DisplayData.Item?)
 - Method in interface