[jira] [Updated] (SAMZA-2067) Samza on Kubernetes
[ https://issues.apache.org/jira/browse/SAMZA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-2067: Fix Version/s: (was: 1.4) 1.7 > Samza on Kubernetes > --- > > Key: SAMZA-2067 > URL: https://issues.apache.org/jira/browse/SAMZA-2067 > Project: Samza > Issue Type: New Feature >Reporter: Weiqing Yang >Priority: Major > Fix For: 1.7 > > Time Spent: 4h > Remaining Estimate: 0h > > This Jira is to support Samza on Kubernetes. > Please refer to the attached document for more details, including the > background, design, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SAMZA-2629) Fix host affinity issue
Weiqing Yang created SAMZA-2629: --- Summary: Fix host affinity issue Key: SAMZA-2629 URL: https://issues.apache.org/jira/browse/SAMZA-2629 Project: Samza Issue Type: Sub-task Reporter: Weiqing Yang In master branch, the getContainerToHostValue() method has been removed from JobModel . Need to figure out how to get the mapping info between Containers and Hosts. May need to do some code refactoring in ContainerProcessManager/ClusterResourceManager. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SAMZA-2067) Samza on Kubernetes
[ https://issues.apache.org/jira/browse/SAMZA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-2067: Fix Version/s: (was: 0.15.0) 1.4 > Samza on Kubernetes > --- > > Key: SAMZA-2067 > URL: https://issues.apache.org/jira/browse/SAMZA-2067 > Project: Samza > Issue Type: New Feature >Reporter: Weiqing Yang >Priority: Major > Fix For: 1.4 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > This Jira is to support Samza on Kubernetes. > Please refer to the attached document for more details, including the > background, design, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SAMZA-2067) Samza on Kubernetes
[ https://issues.apache.org/jira/browse/SAMZA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-2067: Fix Version/s: (was: 1.1) > Samza on Kubernetes > --- > > Key: SAMZA-2067 > URL: https://issues.apache.org/jira/browse/SAMZA-2067 > Project: Samza > Issue Type: New Feature >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > This Jira is to support Samza on Kubernetes. > Please refer to the attached document for more details, including the > background, design, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SAMZA-2368) [Investigate] Figure out "samza.fwk.path" and "samza.fwk.version" are still needed in Samza 1.3
[ https://issues.apache.org/jira/browse/SAMZA-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-2368: Summary: [Investigate] Figure out "samza.fwk.path" and "samza.fwk.version" are still needed in Samza 1.3 (was: [Investigate] Figure out "samza.fwk.path" and "samza.fwk.version" are still needed) > [Investigate] Figure out "samza.fwk.path" and "samza.fwk.version" are still > needed in Samza 1.3 > --- > > Key: SAMZA-2368 > URL: https://issues.apache.org/jira/browse/SAMZA-2368 > Project: Samza > Issue Type: Sub-task >Reporter: Weiqing Yang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SAMZA-2371) Add util methods describing details about Pod status and container status
Weiqing Yang created SAMZA-2371: --- Summary: Add util methods describing details about Pod status and container status Key: SAMZA-2371 URL: https://issues.apache.org/jira/browse/SAMZA-2371 Project: Samza Issue Type: Sub-task Reporter: Weiqing Yang Add util methods describing details about Pod status and container status, then we can invoke these methods in logs and exception messages. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SAMZA-2370) Add unit/integration tests
Weiqing Yang created SAMZA-2370: --- Summary: Add unit/integration tests Key: SAMZA-2370 URL: https://issues.apache.org/jira/browse/SAMZA-2370 Project: Samza Issue Type: Sub-task Reporter: Weiqing Yang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (SAMZA-2369) Add logging thread in KubePodStatusWatcher
[ https://issues.apache.org/jira/browse/SAMZA-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-2369: Parent: SAMZA-2067 Issue Type: Sub-task (was: Bug) > Add logging thread in KubePodStatusWatcher > -- > > Key: SAMZA-2369 > URL: https://issues.apache.org/jira/browse/SAMZA-2369 > Project: Samza > Issue Type: Sub-task >Reporter: Weiqing Yang >Priority: Major > > Please refer to > [LoggingPodStatusWatcher|https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala] > in Spark on K8s. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SAMZA-2369) Add logging thread in KubePodStatusWatcher
Weiqing Yang created SAMZA-2369: --- Summary: Add logging thread in KubePodStatusWatcher Key: SAMZA-2369 URL: https://issues.apache.org/jira/browse/SAMZA-2369 Project: Samza Issue Type: Bug Reporter: Weiqing Yang Please refer to [LoggingPodStatusWatcher|https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala] in Spark on K8s. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SAMZA-2368) [Investigate] Figure out "samza.fwk.path" and "samza.fwk.version" are still needed
Weiqing Yang created SAMZA-2368: --- Summary: [Investigate] Figure out "samza.fwk.path" and "samza.fwk.version" are still needed Key: SAMZA-2368 URL: https://issues.apache.org/jira/browse/SAMZA-2368 Project: Samza Issue Type: Sub-task Reporter: Weiqing Yang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SAMZA-2367) Add label when creating Pod
Weiqing Yang created SAMZA-2367: --- Summary: Add label when creating Pod Key: SAMZA-2367 URL: https://issues.apache.org/jira/browse/SAMZA-2367 Project: Samza Issue Type: Sub-task Reporter: Weiqing Yang In startPodWatcher() in KubeClusterResourceManager, "podLabels" is empty. Need to add label when creating Pod. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SAMZA-2366) [Investigate] should container ID need to be one of the envs for the task container pod
Weiqing Yang created SAMZA-2366: --- Summary: [Investigate] should container ID need to be one of the envs for the task container pod Key: SAMZA-2366 URL: https://issues.apache.org/jira/browse/SAMZA-2366 Project: Samza Issue Type: Sub-task Reporter: Weiqing Yang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SAMZA-2365) Validate KubeConfig before starting the job
Weiqing Yang created SAMZA-2365: --- Summary: Validate KubeConfig before starting the job Key: SAMZA-2365 URL: https://issues.apache.org/jira/browse/SAMZA-2365 Project: Samza Issue Type: Sub-task Reporter: Weiqing Yang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SAMZA-2363) Rock DB failed to create hard link on Azure file
[
https://issues.apache.org/jira/browse/SAMZA-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966916#comment-16966916
]
Weiqing Yang commented on SAMZA-2363:
-
/cc [~xinyu] [~pmaheshwari]
> Rock DB failed to create hard link on Azure file
>
>
> Key: SAMZA-2363
> URL: https://issues.apache.org/jira/browse/SAMZA-2363
> Project: Samza
> Issue Type: Bug
> Components: kv, kv-store
>Reporter: Weiqing Yang
>Priority: Major
>
> I was tried to test the implementation
> ([https://github.com/apache/samza/pull/1197]) of Samza on K8s on AKS (using
> the latest Samza (master branch)), and got the error below. The job was
> trying to store logs and the state in Azure file.
> (P.S. Samza 1.1 does not hit this issue, and the job built on Samza 1.1 works
> fine on AKS.)
> {code:java}
> 2019-11-04 00:56:00.558 [main] SamzaContainer [INFO] Entering run loop.
> 2019-11-04 00:56:00.559 [main] ClusterBasedProcessorLifecycleListener [INFO]
> Container Started
> 2019-11-04 00:56:57.106 [kafka-producer-network-thread |
> kafka_producer-wikipedia_application-1] Metadata [INFO] Cluster ID:
> YNWIzBLFSa2Lg5wehzo7ZA
> 2019-11-04 00:57:03.920 [main] KafkaSystemAdmin [INFO] Fetching SSP metadata
> for: [SystemStreamPartition [kafka, wikipedia-stats-changelog, 7]]
> 2019-11-04 00:57:03.923 [main] KafkaSystemAdmin [INFO] Fetching SSP metadata
> for: [SystemStreamPartition [kafka,
> wikipedia-application-1-window-statsWindow, 7]]
> 2019-11-04 00:57:04.222 [main] RunLoop [ERROR] Task Partition 7 commit failed
> org.rocksdb.RocksDBException: while link file to
> /tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7-1572829023927.tmp/13.sst:
> /tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7/13.sst:
> Operation not supported
> at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
> at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51)
> at
> org.apache.samza.storage.kv.RocksDbKeyValueStore.checkpoint(RocksDbKeyValueStore.scala:245)
> at
> org.apache.samza.storage.kv.LoggedStore.checkpoint(LoggedStore.scala:125)
> at
> org.apache.samza.storage.kv.SerializedKeyValueStore.checkpoint(SerializedKeyValueStore.scala:170)
> at
> org.apache.samza.storage.kv.CachedStore.checkpoint(CachedStore.scala:297)
> at
> org.apache.samza.storage.kv.NullSafeKeyValueStore.checkpoint(NullSafeKeyValueStore.scala:103)
> at
> org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$checkpoint$1.apply(KeyValueStorageEngine.scala:215)
> at
> org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$checkpoint$1.apply(KeyValueStorageEngine.scala:212)
> at org.apache.samza.util.TimerUtil$class.updateTimer(TimerUtil.scala:37)
> at
> org.apache.samza.storage.kv.KeyValueStorageEngine.updateTimer(KeyValueStorageEngine.scala:37)
> at
> org.apache.samza.storage.kv.KeyValueStorageEngine.checkpoint(KeyValueStorageEngine.scala:212)
> at
> org.apache.samza.storage.TransactionalStateTaskStorageManager$$anonfun$2.apply(TransactionalStateTaskStorageManager.scala:67)
> at
> org.apache.samza.storage.TransactionalStateTaskStorageManager$$anonfun$2.apply(TransactionalStateTaskStorageManager.scala:66)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
> at
> org.apache.samza.storage.TransactionalStateTaskStorageManager.checkpoint(TransactionalStateTaskStorageManager.scala:66)
> at
> org.apache.samza.container.TaskInstance.commit(TaskInstance.scala:267)
> at
> org.apache.samza.container.RunLoop$AsyncTaskWorker$5.run(RunLoop.java:547)
> at
> org.apache.samza.container.RunLoop$AsyncTaskWorker.commit(RunLoop.java:566)
> at
> org.apache.samza.container.RunLoop$AsyncTaskWorker.run(RunLoop.java:432)
> at
> org.apache.samza.container.RunLoop$AsyncTaskWorker.access$300(RunLoop.java:357)
> at org.apache.samza.container.RunLoop.runTasks(RunLoop.java:244)
> at org.apache.samza.container.RunLoop.run(RunLoop.java:176)
> at
> org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:768)
> at
> org.apache.samza.runtime.ContainerLaunchUtil.run(ContainerLaunchUtil.java:139)
>
[jira] [Updated] (SAMZA-2363) Rock DB failed to create hard link on Azure file
[
https://issues.apache.org/jira/browse/SAMZA-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiqing Yang updated SAMZA-2363:
Description:
I was tried to test the implementation
([https://github.com/apache/samza/pull/1197]) of Samza on K8s on AKS (using the
latest Samza (master branch)), and got the error below. The job was trying to
store logs and the state in Azure file.
(P.S. Samza 1.1 does not hit this issue, and the job built on Samza 1.1 works
fine on AKS.)
{code:java}
2019-11-04 00:56:00.558 [main] SamzaContainer [INFO] Entering run loop.
2019-11-04 00:56:00.559 [main] ClusterBasedProcessorLifecycleListener [INFO]
Container Started
2019-11-04 00:56:57.106 [kafka-producer-network-thread |
kafka_producer-wikipedia_application-1] Metadata [INFO] Cluster ID:
YNWIzBLFSa2Lg5wehzo7ZA
2019-11-04 00:57:03.920 [main] KafkaSystemAdmin [INFO] Fetching SSP metadata
for: [SystemStreamPartition [kafka, wikipedia-stats-changelog, 7]]
2019-11-04 00:57:03.923 [main] KafkaSystemAdmin [INFO] Fetching SSP metadata
for: [SystemStreamPartition [kafka, wikipedia-application-1-window-statsWindow,
7]]
2019-11-04 00:57:04.222 [main] RunLoop [ERROR] Task Partition 7 commit failed
org.rocksdb.RocksDBException: while link file to
/tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7-1572829023927.tmp/13.sst:
/tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7/13.sst:
Operation not supported
at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51)
at
org.apache.samza.storage.kv.RocksDbKeyValueStore.checkpoint(RocksDbKeyValueStore.scala:245)
at
org.apache.samza.storage.kv.LoggedStore.checkpoint(LoggedStore.scala:125)
at
org.apache.samza.storage.kv.SerializedKeyValueStore.checkpoint(SerializedKeyValueStore.scala:170)
at
org.apache.samza.storage.kv.CachedStore.checkpoint(CachedStore.scala:297)
at
org.apache.samza.storage.kv.NullSafeKeyValueStore.checkpoint(NullSafeKeyValueStore.scala:103)
at
org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$checkpoint$1.apply(KeyValueStorageEngine.scala:215)
at
org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$checkpoint$1.apply(KeyValueStorageEngine.scala:212)
at org.apache.samza.util.TimerUtil$class.updateTimer(TimerUtil.scala:37)
at
org.apache.samza.storage.kv.KeyValueStorageEngine.updateTimer(KeyValueStorageEngine.scala:37)
at
org.apache.samza.storage.kv.KeyValueStorageEngine.checkpoint(KeyValueStorageEngine.scala:212)
at
org.apache.samza.storage.TransactionalStateTaskStorageManager$$anonfun$2.apply(TransactionalStateTaskStorageManager.scala:67)
at
org.apache.samza.storage.TransactionalStateTaskStorageManager$$anonfun$2.apply(TransactionalStateTaskStorageManager.scala:66)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at
org.apache.samza.storage.TransactionalStateTaskStorageManager.checkpoint(TransactionalStateTaskStorageManager.scala:66)
at
org.apache.samza.container.TaskInstance.commit(TaskInstance.scala:267)
at
org.apache.samza.container.RunLoop$AsyncTaskWorker$5.run(RunLoop.java:547)
at
org.apache.samza.container.RunLoop$AsyncTaskWorker.commit(RunLoop.java:566)
at
org.apache.samza.container.RunLoop$AsyncTaskWorker.run(RunLoop.java:432)
at
org.apache.samza.container.RunLoop$AsyncTaskWorker.access$300(RunLoop.java:357)
at org.apache.samza.container.RunLoop.runTasks(RunLoop.java:244)
at org.apache.samza.container.RunLoop.run(RunLoop.java:176)
at
org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:768)
at
org.apache.samza.runtime.ContainerLaunchUtil.run(ContainerLaunchUtil.java:139)
at
org.apache.samza.runtime.ContainerLaunchUtil.run(ContainerLaunchUtil.java:82)
at
org.apache.samza.runtime.LocalContainerRunner.main(LocalContainerRunner.java:80)
2019-11-04 00:57:04.223 [main] RunLoop [ERROR] Caught throwable and stopping
run loop
org.rocksdb.RocksDBException: while link file to
/tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7-1572829023927.tmp/13.sst:
/tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7/13.sst:
Operation not supported
[jira] [Created] (SAMZA-2363) Rock DB failed to create hard link on Azure file
Weiqing Yang created SAMZA-2363:
---
Summary: Rock DB failed to create hard link on Azure file
Key: SAMZA-2363
URL: https://issues.apache.org/jira/browse/SAMZA-2363
Project: Samza
Issue Type: Bug
Components: kv, kv-store
Reporter: Weiqing Yang
I was tried to test the implementation
([https://github.com/apache/samza/pull/1197]) of Samza on K8s on AKS (using the
latest Samza (master branch)), and got the error below. The job was trying to
store logs and the state in Azure file.
(P.S. the implementation works fine with Samza 1.1.)
{code:java}
2019-11-04 00:56:00.558 [main] SamzaContainer [INFO] Entering run loop.
2019-11-04 00:56:00.559 [main] ClusterBasedProcessorLifecycleListener [INFO]
Container Started
2019-11-04 00:56:57.106 [kafka-producer-network-thread |
kafka_producer-wikipedia_application-1] Metadata [INFO] Cluster ID:
YNWIzBLFSa2Lg5wehzo7ZA
2019-11-04 00:57:03.920 [main] KafkaSystemAdmin [INFO] Fetching SSP metadata
for: [SystemStreamPartition [kafka, wikipedia-stats-changelog, 7]]
2019-11-04 00:57:03.923 [main] KafkaSystemAdmin [INFO] Fetching SSP metadata
for: [SystemStreamPartition [kafka, wikipedia-application-1-window-statsWindow,
7]]
2019-11-04 00:57:04.222 [main] RunLoop [ERROR] Task Partition 7 commit failed
org.rocksdb.RocksDBException: while link file to
/tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7-1572829023927.tmp/13.sst:
/tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7/13.sst:
Operation not supported
at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51)
at
org.apache.samza.storage.kv.RocksDbKeyValueStore.checkpoint(RocksDbKeyValueStore.scala:245)
at
org.apache.samza.storage.kv.LoggedStore.checkpoint(LoggedStore.scala:125)
at
org.apache.samza.storage.kv.SerializedKeyValueStore.checkpoint(SerializedKeyValueStore.scala:170)
at
org.apache.samza.storage.kv.CachedStore.checkpoint(CachedStore.scala:297)
at
org.apache.samza.storage.kv.NullSafeKeyValueStore.checkpoint(NullSafeKeyValueStore.scala:103)
at
org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$checkpoint$1.apply(KeyValueStorageEngine.scala:215)
at
org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$checkpoint$1.apply(KeyValueStorageEngine.scala:212)
at org.apache.samza.util.TimerUtil$class.updateTimer(TimerUtil.scala:37)
at
org.apache.samza.storage.kv.KeyValueStorageEngine.updateTimer(KeyValueStorageEngine.scala:37)
at
org.apache.samza.storage.kv.KeyValueStorageEngine.checkpoint(KeyValueStorageEngine.scala:212)
at
org.apache.samza.storage.TransactionalStateTaskStorageManager$$anonfun$2.apply(TransactionalStateTaskStorageManager.scala:67)
at
org.apache.samza.storage.TransactionalStateTaskStorageManager$$anonfun$2.apply(TransactionalStateTaskStorageManager.scala:66)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at
org.apache.samza.storage.TransactionalStateTaskStorageManager.checkpoint(TransactionalStateTaskStorageManager.scala:66)
at
org.apache.samza.container.TaskInstance.commit(TaskInstance.scala:267)
at
org.apache.samza.container.RunLoop$AsyncTaskWorker$5.run(RunLoop.java:547)
at
org.apache.samza.container.RunLoop$AsyncTaskWorker.commit(RunLoop.java:566)
at
org.apache.samza.container.RunLoop$AsyncTaskWorker.run(RunLoop.java:432)
at
org.apache.samza.container.RunLoop$AsyncTaskWorker.access$300(RunLoop.java:357)
at org.apache.samza.container.RunLoop.runTasks(RunLoop.java:244)
at org.apache.samza.container.RunLoop.run(RunLoop.java:176)
at
org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:768)
at
org.apache.samza.runtime.ContainerLaunchUtil.run(ContainerLaunchUtil.java:139)
at
org.apache.samza.runtime.ContainerLaunchUtil.run(ContainerLaunchUtil.java:82)
at
org.apache.samza.runtime.LocalContainerRunner.main(LocalContainerRunner.java:80)
2019-11-04 00:57:04.223 [main] RunLoop [ERROR] Caught throwable and stopping
run loop
org.rocksdb.RocksDBException: while link file to
[jira] [Commented] (SAMZA-2067) Samza on Kubernetes
[ https://issues.apache.org/jira/browse/SAMZA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16890456#comment-16890456 ] Weiqing Yang commented on SAMZA-2067: - This issue is WIP > Samza on Kubernetes > --- > > Key: SAMZA-2067 > URL: https://issues.apache.org/jira/browse/SAMZA-2067 > Project: Samza > Issue Type: New Feature >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0, 1.1 > > > This Jira is to support Samza on Kubernetes. > Please refer to the attached document for more details, including the > background, design, etc. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (SAMZA-1960) Refactor QueryTranslator unit tests
[ https://issues.apache.org/jira/browse/SAMZA-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16890455#comment-16890455 ] Weiqing Yang commented on SAMZA-1960: - The issue 2 has been resolved. > Refactor QueryTranslator unit tests > --- > > Key: SAMZA-1960 > URL: https://issues.apache.org/jira/browse/SAMZA-1960 > Project: Samza > Issue Type: Bug > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > This Jira is to track the follow issues: > # Currently, the unit tests about "QueryTranslator" are using their own > version of > [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107](), > the logic of which is a little messy. We should either use the translate() > [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] > or refactor the test version of > [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107]() > and its related unit tests. > # We should rename the package name of > "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; > to "util". -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (SAMZA-1901) Samza SQL Shell
[ https://issues.apache.org/jira/browse/SAMZA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang resolved SAMZA-1901. - Resolution: Fixed > Samza SQL Shell > --- > > Key: SAMZA-1901 > URL: https://issues.apache.org/jira/browse/SAMZA-1901 > Project: Samza > Issue Type: New Feature > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > This Jira implements the first version of Samza sql shell. > Please refer to the attached document for more details about the shell, > including the tech choices, features, design decisions, how to build, run and > debug the shell. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (SAMZA-2247) Fix errors when getting pod status using Watcher
[ https://issues.apache.org/jira/browse/SAMZA-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-2247: Issue Type: Sub-task (was: Bug) Parent: SAMZA-2067 > Fix errors when getting pod status using Watcher > > > Key: SAMZA-2247 > URL: https://issues.apache.org/jira/browse/SAMZA-2247 > Project: Samza > Issue Type: Sub-task > Environment: Client can not exit when using Watcher to monitor pod > status. This issue could be K8s Watcher issue. >Reporter: Weiqing Yang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-2247) Fix errors when getting pod status using Watcher
Weiqing Yang created SAMZA-2247: --- Summary: Fix errors when getting pod status using Watcher Key: SAMZA-2247 URL: https://issues.apache.org/jira/browse/SAMZA-2247 Project: Samza Issue Type: Bug Environment: Client can not exit when using Watcher to monitor pod status. This issue could be K8s Watcher issue. Reporter: Weiqing Yang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (SAMZA-2005) Validate Sql
[ https://issues.apache.org/jira/browse/SAMZA-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang reopened SAMZA-2005: - > Validate Sql > > > Key: SAMZA-2005 > URL: https://issues.apache.org/jira/browse/SAMZA-2005 > Project: Samza > Issue Type: Sub-task >Reporter: Weiqing Yang >Priority: Major > > We can implement Samza SQL validation: Syntax, resource presence, schema > presence, etc. (including unit tests with Samza SQL test framework) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SAMZA-1902) Support different data systems
[ https://issues.apache.org/jira/browse/SAMZA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang resolved SAMZA-1902. - Resolution: Fixed > Support different data systems > -- > > Key: SAMZA-1902 > URL: https://issues.apache.org/jira/browse/SAMZA-1902 > Project: Samza > Issue Type: Sub-task >Reporter: Weiqing Yang >Priority: Major > > Currently the Shell can only talk to Kafka system, but we may need to use a > general way to connect to different systems. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SAMZA-2027) Config support in through Set statement
[ https://issues.apache.org/jira/browse/SAMZA-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang resolved SAMZA-2027. - Resolution: Fixed > Config support in through Set statement > --- > > Key: SAMZA-2027 > URL: https://issues.apache.org/jira/browse/SAMZA-2027 > Project: Samza > Issue Type: Sub-task > Components: sql >Reporter: Weiqing Yang >Priority: Major > > Supports environment variables for SqlExecutors. User can now set environment > variables for the shell itself and for any SqlExecutor, by executing the > 'SET' command in the shell or by using a configuration file. User can also > choose to use any SqlExecutor they want in the same way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SAMZA-2005) Validate Sql
[ https://issues.apache.org/jira/browse/SAMZA-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang resolved SAMZA-2005. - Resolution: Fixed > Validate Sql > > > Key: SAMZA-2005 > URL: https://issues.apache.org/jira/browse/SAMZA-2005 > Project: Samza > Issue Type: Sub-task >Reporter: Weiqing Yang >Priority: Major > > We can implement Samza SQL validation: Syntax, resource presence, schema > presence, etc. (including unit tests with Samza SQL test framework) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (SAMZA-1901) Samza SQL Shell
[ https://issues.apache.org/jira/browse/SAMZA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang reopened SAMZA-1901: - > Samza SQL Shell > --- > > Key: SAMZA-1901 > URL: https://issues.apache.org/jira/browse/SAMZA-1901 > Project: Samza > Issue Type: New Feature > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > This Jira implements the first version of Samza sql shell. > Please refer to the attached document for more details about the shell, > including the tech choices, features, design decisions, how to build, run and > debug the shell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SAMZA-1901) Samza SQL Shell
[ https://issues.apache.org/jira/browse/SAMZA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang resolved SAMZA-1901. - Resolution: Fixed > Samza SQL Shell > --- > > Key: SAMZA-1901 > URL: https://issues.apache.org/jira/browse/SAMZA-1901 > Project: Samza > Issue Type: New Feature > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > This Jira implements the first version of Samza sql shell. > Please refer to the attached document for more details about the shell, > including the tech choices, features, design decisions, how to build, run and > debug the shell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-2096) Support getVersion in Sql Shell
Weiqing Yang created SAMZA-2096: --- Summary: Support getVersion in Sql Shell Key: SAMZA-2096 URL: https://issues.apache.org/jira/browse/SAMZA-2096 Project: Samza Issue Type: Improvement Components: sql Reporter: Weiqing Yang This Jira is to support getting version information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-2093) Make Sql Shell not depend on result schemas
Weiqing Yang created SAMZA-2093: --- Summary: Make Sql Shell not depend on result schemas Key: SAMZA-2093 URL: https://issues.apache.org/jira/browse/SAMZA-2093 Project: Samza Issue Type: Bug Components: sql Reporter: Weiqing Yang After SAMZA-2079 was fixed, we can revert the PR of this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-2084) Samza Sql: add logging when exceptions are swallowed and rename util package
Weiqing Yang created SAMZA-2084: --- Summary: Samza Sql: add logging when exceptions are swallowed and rename util package Key: SAMZA-2084 URL: https://issues.apache.org/jira/browse/SAMZA-2084 Project: Samza Issue Type: Bug Components: sql Reporter: Weiqing Yang Fix For: 0.15, 1.1 This Jira is to track 1) Samza Sql Shell: add logging when exceptions are swallowed 2) Samza Sql: rename the package name of "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; to "util". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SAMZA-2067) Samza on Kubernetes
[ https://issues.apache.org/jira/browse/SAMZA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742842#comment-16742842 ] Weiqing Yang commented on SAMZA-2067: - + [~jianhe] > Samza on Kubernetes > --- > > Key: SAMZA-2067 > URL: https://issues.apache.org/jira/browse/SAMZA-2067 > Project: Samza > Issue Type: New Feature >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0, 1.1 > > > This Jira is to support Samza on Kubernetes. > Please refer to the attached document for more details, including the > background, design, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-2067) Samza on Kubernetes
Weiqing Yang created SAMZA-2067: --- Summary: Samza on Kubernetes Key: SAMZA-2067 URL: https://issues.apache.org/jira/browse/SAMZA-2067 Project: Samza Issue Type: New Feature Reporter: Weiqing Yang This Jira is to support Samza on Kubernetes. Please refer to the attached document for more details, including the background, design, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-2067) Samza on Kubernetes
[ https://issues.apache.org/jira/browse/SAMZA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-2067: Fix Version/s: 1.1 0.15.0 > Samza on Kubernetes > --- > > Key: SAMZA-2067 > URL: https://issues.apache.org/jira/browse/SAMZA-2067 > Project: Samza > Issue Type: New Feature >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0, 1.1 > > > This Jira is to support Samza on Kubernetes. > Please refer to the attached document for more details, including the > background, design, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-2027) Config support in through Set statement
Weiqing Yang created SAMZA-2027: --- Summary: Config support in through Set statement Key: SAMZA-2027 URL: https://issues.apache.org/jira/browse/SAMZA-2027 Project: Samza Issue Type: Sub-task Components: sql Reporter: Weiqing Yang Supports environment variables for SqlExecutors. User can now set environment variables for the shell itself and for any SqlExecutor, by executing the 'SET' command in the shell or by using a configuration file. User can also choose to use any SqlExecutor they want in the same way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-2005) Validate Sql
[ https://issues.apache.org/jira/browse/SAMZA-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-2005: Description: We can implement Samza SQL validation: Syntax, resource presence, schema presence, etc. (including unit tests with Samza SQL test framework) (was: We can implement Samza SQL validation: Syntax, resource presence, schema presence, etc.) > Validate Sql > > > Key: SAMZA-2005 > URL: https://issues.apache.org/jira/browse/SAMZA-2005 > Project: Samza > Issue Type: Sub-task >Reporter: Weiqing Yang >Priority: Major > > We can implement Samza SQL validation: Syntax, resource presence, schema > presence, etc. (including unit tests with Samza SQL test framework) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-2005) Validate Sql
Weiqing Yang created SAMZA-2005: --- Summary: Validate Sql Key: SAMZA-2005 URL: https://issues.apache.org/jira/browse/SAMZA-2005 Project: Samza Issue Type: Sub-task Reporter: Weiqing Yang We can implement Samza SQL validation: Syntax, resource presence, schema presence, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-2003) Clean logic about logging system and add test cases
Weiqing Yang created SAMZA-2003: --- Summary: Clean logic about logging system and add test cases Key: SAMZA-2003 URL: https://issues.apache.org/jira/browse/SAMZA-2003 Project: Samza Issue Type: Bug Components: sql Reporter: Weiqing Yang This Jira is to track: 1. Clean logic about logging system. We may want to remove the workaround like in SAMZA-1999. 2. Add test cases for logging system, which is critical for Samza SQL Shell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-1999) Throw NullPointerException when sink is log.outputstream
Weiqing Yang created SAMZA-1999: --- Summary: Throw NullPointerException when sink is log.outputstream Key: SAMZA-1999 URL: https://issues.apache.org/jira/browse/SAMZA-1999 Project: Samza Issue Type: Bug Components: sql Reporter: Weiqing Yang This Jira is to fix the NullPointerException when sink is log.outputstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-1901) Samza SQL Shell
[ https://issues.apache.org/jira/browse/SAMZA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-1901: Attachment: (was: Samza SQL Shell Overview.docx) > Samza SQL Shell > --- > > Key: SAMZA-1901 > URL: https://issues.apache.org/jira/browse/SAMZA-1901 > Project: Samza > Issue Type: New Feature > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > This Jira implements the first version of Samza sql shell. > Please refer to the attached document for more details about the shell, > including the tech choices, features, design decisions, how to build, run and > debug the shell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SAMZA-1960) Refactor QueryTranslator unit tests
[ https://issues.apache.org/jira/browse/SAMZA-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652713#comment-16652713 ] Weiqing Yang commented on SAMZA-1960: - [@srinipunuru|https://github.com/srinipunuru] [@atoomula|https://github.com/atoomula] > Refactor QueryTranslator unit tests > --- > > Key: SAMZA-1960 > URL: https://issues.apache.org/jira/browse/SAMZA-1960 > Project: Samza > Issue Type: Bug > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > This Jira is to track the follow issues: > # Currently, the unit tests about "QueryTranslator" are using their own > version of > [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107](), > the logic of which is a little messy. We should either use the translate() > [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] > or refactor the test version of > [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107]() > and its related unit tests. > # We should rename the package name of > "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; > to "util". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-1960) Refactor QueryTranslator unit tests
[ https://issues.apache.org/jira/browse/SAMZA-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-1960: Description: This Jira is to track the follow issues: # Currently, the unit tests about "QueryTranslator" are using their own version of [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107](), the logic of which is a little messy. We should either use the translate() [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] or refactor the test version of [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107]() and its related unit tests. # We should rename the package name of "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; to "util". was: This Jira is to tracking the follow issues: # Currently, the unit tests about "QueryTranslator" are using their own version of [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107](), the logic of which is a little messy. We should either use the translate() [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] or refactor the test version of [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107]() and its related unit tests. # We should rename the package name of "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; to "util". > Refactor QueryTranslator unit tests > --- > > Key: SAMZA-1960 > URL: https://issues.apache.org/jira/browse/SAMZA-1960 > Project: Samza > Issue Type: Bug > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > This Jira is to track the follow issues: > # Currently, the unit tests about "QueryTranslator" are using their own > version of > [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107](), > the logic of which is a little messy. We should either use the translate() > [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] > or refactor the test version of > [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107]() > and its related unit tests. > # We should rename the package name of > "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; > to "util". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-1960) Refactor QueryTranslator unit tests
[ https://issues.apache.org/jira/browse/SAMZA-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-1960: Description: This Jira is to tracking the follow issues: # Currently, the unit tests about "QueryTranslator" are using their own version of [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107](), the logic of which is a little messy. We should either use the translate() [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] or refactor the test version of [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107]() and its related unit tests. # We should rename the package name of "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; to "util". was: This Jira is to tracking the follow issues: # Currently, the unit tests about QueryTranslator are using their own version of [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107](), whose logic is a little messy. We should either use the translate() [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] or refactor it and its related unit tests. # We should rename the package name of "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; to "util". > Refactor QueryTranslator unit tests > --- > > Key: SAMZA-1960 > URL: https://issues.apache.org/jira/browse/SAMZA-1960 > Project: Samza > Issue Type: Bug > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > This Jira is to tracking the follow issues: > # Currently, the unit tests about "QueryTranslator" are using their own > version of > [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107](), > the logic of which is a little messy. We should either use the translate() > [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] > or refactor the test version of > [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107]() > and its related unit tests. > # We should rename the package name of > "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; > to "util". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-1960) Refactor QueryTranslator unit tests
[ https://issues.apache.org/jira/browse/SAMZA-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-1960: Description: This Jira is to tracking the follow issues: # Currently, the unit tests about QueryTranslator are using their own version of [translate()| [https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107] ], whose logic is a little messy. We should either use the translate() [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] or refactor it and its related unit tests. # We should rename the package name of "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; to "util". was: This Jira is to tracking the follow issues: # Currently, the unit tests about QueryTranslator are using their own version of [translate()|[https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107]|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107],], whose logic is a little messy. We should either use the translate() [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] or refactor it and its related unit tests. # We should rename the package name of "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; to "util". > Refactor QueryTranslator unit tests > --- > > Key: SAMZA-1960 > URL: https://issues.apache.org/jira/browse/SAMZA-1960 > Project: Samza > Issue Type: Bug > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > This Jira is to tracking the follow issues: > # Currently, the unit tests about QueryTranslator are using their own > version of [translate()| > [https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107] > ], whose logic is a little messy. We should either use the translate() > [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] > or refactor it and its related unit tests. > # We should rename the package name of > "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; > to "util". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-1960) Refactor QueryTranslator unit tests
[ https://issues.apache.org/jira/browse/SAMZA-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-1960: Description: This Jira is to tracking the follow issues: # Currently, the unit tests about QueryTranslator are using their own version of [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107](), whose logic is a little messy. We should either use the translate() [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] or refactor it and its related unit tests. # We should rename the package name of "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; to "util". was: This Jira is to tracking the follow issues: # Currently, the unit tests about QueryTranslator are using their own version of [translate()| [https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107] ], whose logic is a little messy. We should either use the translate() [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] or refactor it and its related unit tests. # We should rename the package name of "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; to "util". > Refactor QueryTranslator unit tests > --- > > Key: SAMZA-1960 > URL: https://issues.apache.org/jira/browse/SAMZA-1960 > Project: Samza > Issue Type: Bug > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > This Jira is to tracking the follow issues: > # Currently, the unit tests about QueryTranslator are using their own > version of > [translate|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107](), > whose logic is a little messy. We should either use the translate() > [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] > or refactor it and its related unit tests. > # We should rename the package name of > "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; > to "util". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-1960) Refactor QueryTranslator unit tests
Weiqing Yang created SAMZA-1960: --- Summary: Refactor QueryTranslator unit tests Key: SAMZA-1960 URL: https://issues.apache.org/jira/browse/SAMZA-1960 Project: Samza Issue Type: Bug Components: sql Reporter: Weiqing Yang Fix For: 0.15.0 This Jira is to tracking the follow issues: # Currently, the unit tests about QueryTranslator are using their own version of [translate()|[https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107]|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L107],], whose logic is a little messy. We should either use the translate() [here|https://github.com/apache/samza/blob/master/samza-sql/src/main/java/org/apache/samza/sql/translator/QueryTranslator.java#L126] or refactor it and its related unit tests. # We should rename the package name of "[testutil|https://github.com/apache/samza/tree/master/samza-sql/src/main/java/org/apache/samza/sql/testutil]; to "util". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SAMZA-1958) Add an API in ScalarUdf to get the Udf display information
[ https://issues.apache.org/jira/browse/SAMZA-1958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang closed SAMZA-1958. --- Duplicate > Add an API in ScalarUdf to get the Udf display information > -- > > Key: SAMZA-1958 > URL: https://issues.apache.org/jira/browse/SAMZA-1958 > Project: Samza > Issue Type: Sub-task > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > Currently if users implement new UDFs or if Samza SQL implements a new system > UDF, they need to manually register that UDF's information in the > "listFunctions" of Samza executor. So we may need to require UDFs to provide > a function of getting their "SamzaSqlUdfDisplayInfo", then Samza executor can > get the UDF information from SamzaSqlApplicationConfig.udfResolver(or > SamzaSqlApplicationConfig.udfMetadata) instead of registering UDFs manually. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SAMZA-1958) Add an API in ScalarUdf to get the Udf display information
[ https://issues.apache.org/jira/browse/SAMZA-1958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang resolved SAMZA-1958. - Resolution: Duplicate > Add an API in ScalarUdf to get the Udf display information > -- > > Key: SAMZA-1958 > URL: https://issues.apache.org/jira/browse/SAMZA-1958 > Project: Samza > Issue Type: Sub-task > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > Currently if users implement new UDFs or if Samza SQL implements a new system > UDF, they need to manually register that UDF's information in the > "listFunctions" of Samza executor. So we may need to require UDFs to provide > a function of getting their "SamzaSqlUdfDisplayInfo", then Samza executor can > get the UDF information from SamzaSqlApplicationConfig.udfResolver(or > SamzaSqlApplicationConfig.udfMetadata) instead of registering UDFs manually. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-1957) Add an API in ScalarUdf to get the Udf display information
[ https://issues.apache.org/jira/browse/SAMZA-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-1957: Issue Type: Sub-task (was: Test) Parent: SAMZA-1901 > Add an API in ScalarUdf to get the Udf display information > -- > > Key: SAMZA-1957 > URL: https://issues.apache.org/jira/browse/SAMZA-1957 > Project: Samza > Issue Type: Sub-task > Components: sql >Reporter: Weiqing Yang >Priority: Major > > Currently if users implement new UDFs or if Samza SQL implements a new system > UDF, they need to manually register that UDF's information in the > "listFunctions" of Samza executor. So we may need to require UDFs to provide > a function of getting their "SamzaSqlUdfDisplayInfo", then Samza executor can > get the UDF information from SamzaSqlApplicationConfig.udfResolver(or > SamzaSqlApplicationConfig.udfMetadata) instead of registering UDFs manually. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-1958) Add an API in ScalarUdf to get the Udf display information
Weiqing Yang created SAMZA-1958: --- Summary: Add an API in ScalarUdf to get the Udf display information Key: SAMZA-1958 URL: https://issues.apache.org/jira/browse/SAMZA-1958 Project: Samza Issue Type: Sub-task Components: sql Reporter: Weiqing Yang Fix For: 0.15.0 Currently if users implement new UDFs or if Samza SQL implements a new system UDF, they need to manually register that UDF's information in the "listFunctions" of Samza executor. So we may need to require UDFs to provide a function of getting their "SamzaSqlUdfDisplayInfo", then Samza executor can get the UDF information from SamzaSqlApplicationConfig.udfResolver(or SamzaSqlApplicationConfig.udfMetadata) instead of registering UDFs manually. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-1957) Add an API in ScalarUdf to get the Udf display information
Weiqing Yang created SAMZA-1957: --- Summary: Add an API in ScalarUdf to get the Udf display information Key: SAMZA-1957 URL: https://issues.apache.org/jira/browse/SAMZA-1957 Project: Samza Issue Type: Test Components: sql Reporter: Weiqing Yang Currently if users implement new UDFs or if Samza SQL implements a new system UDF, they need to manually register that UDF's information in the "listFunctions" of Samza executor. So we may need to require UDFs to provide a function of getting their "SamzaSqlUdfDisplayInfo", then Samza executor can get the UDF information from SamzaSqlApplicationConfig.udfResolver(or SamzaSqlApplicationConfig.udfMetadata) instead of registering UDFs manually. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SAMZA-1937) Enable users to perform Sql queries (SELECT statement)
[ https://issues.apache.org/jira/browse/SAMZA-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang resolved SAMZA-1937. - Resolution: Fixed > Enable users to perform Sql queries (SELECT statement) > -- > > Key: SAMZA-1937 > URL: https://issues.apache.org/jira/browse/SAMZA-1937 > Project: Samza > Issue Type: Bug > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > > Current DSL converter does not support the use case that users use Samza Sql > Console or Samza Sql shell to perform SELECT query. The reason is > "log.outputStream" does not have schema while Calcite asks for its schema > when Calcite do validation for the Sql statement (we only passed selectQuery > to Calcite before, but now we pass the whole sql to Calcite). Also, it seems > "log.outputStream" does not work with ModifyTranslator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-1938) Support use cases to run multiple sql statements in one Samza job
Weiqing Yang created SAMZA-1938: --- Summary: Support use cases to run multiple sql statements in one Samza job Key: SAMZA-1938 URL: https://issues.apache.org/jira/browse/SAMZA-1938 Project: Samza Issue Type: Bug Components: sql Reporter: Weiqing Yang Fix For: 0.15.0 This Jira is to track the following tasks: * Support users to run multiple Sql statements in the same job. ** sql1 = "Insert into testavro.simpleOutputTopic select * from testavro.SIMPLE1"; sql2 = "Insert into testavro.SIMPLE3 select * from testavro.SIMPLE2"; * Support fan-out use case. For example, ** sql1 = "Insert into testavro.SIMPLE2 select * from testavro.SIMPLE1"; sql2 = "Insert into testavro.SIMPLE3 select * from testavro.SIMPLE1"; * Support fan-in use case. For example, ** sql1 = "Insert into testavro.simpleOutputTopic select * from testavro.SIMPLE2"; sql2 = "Insert into testavro.simpleOutputTopic select * from testavro.SIMPLE1"; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-1937) Enable users to perform Sql queries (SELECT statement)
Weiqing Yang created SAMZA-1937: --- Summary: Enable users to perform Sql queries (SELECT statement) Key: SAMZA-1937 URL: https://issues.apache.org/jira/browse/SAMZA-1937 Project: Samza Issue Type: Bug Components: sql Reporter: Weiqing Yang Fix For: 0.15.0 Current DSL converter does not support the use case that users use Samza Sql Console or Samza Sql shell to perform SELECT query. The reason is "log.outputStream" does not have schema while Calcite asks for its schema when Calcite do validation for the Sql statement (we only passed selectQuery to Calcite before, but now we pass the whole sql to Calcite). Also, it seems "log.outputStream" does not work with ModifyTranslator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-1902) Support different data systems
[ https://issues.apache.org/jira/browse/SAMZA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-1902: Description: Currently the Shell can only talk to Kafka system, but we may need to use a general way to connect to different systems. (was: currently the Shell can only talk to Kafka system, but we may need to use a general way to connect to different systems.) > Support different data systems > -- > > Key: SAMZA-1902 > URL: https://issues.apache.org/jira/browse/SAMZA-1902 > Project: Samza > Issue Type: Sub-task >Reporter: Weiqing Yang >Priority: Major > > Currently the Shell can only talk to Kafka system, but we may need to use a > general way to connect to different systems. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-1902) Support different data systems
Weiqing Yang created SAMZA-1902: --- Summary: Support different data systems Key: SAMZA-1902 URL: https://issues.apache.org/jira/browse/SAMZA-1902 Project: Samza Issue Type: Sub-task Reporter: Weiqing Yang currently the Shell can only talk to Kafka system, but we may need to use a general way to connect to different systems. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-1901) Samza SQL Shell
[ https://issues.apache.org/jira/browse/SAMZA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-1901: Attachment: Samza SQL Shell Overview.docx > Samza SQL Shell > --- > > Key: SAMZA-1901 > URL: https://issues.apache.org/jira/browse/SAMZA-1901 > Project: Samza > Issue Type: New Feature > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > Attachments: Samza SQL Shell Overview.docx > > > This Jira implements the first version of Samza sql shell. > Please refer to the attached document for more details about the shell, > including the tech choices, features, design decisions, how to build, run and > debug the shell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-1901) Samza SQL Shell
[ https://issues.apache.org/jira/browse/SAMZA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-1901: Attachment: (was: Samza SQL Shell Overview.docx) > Samza SQL Shell > --- > > Key: SAMZA-1901 > URL: https://issues.apache.org/jira/browse/SAMZA-1901 > Project: Samza > Issue Type: New Feature > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > Attachments: Samza SQL Shell Overview.docx > > > This Jira implements the first version of Samza sql shell. > Please refer to the attached document for more details about the shell, > including the tech choices, features, design decisions, how to build, run and > debug the shell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-1901) Samza SQL Shell
[ https://issues.apache.org/jira/browse/SAMZA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SAMZA-1901: Attachment: Samza SQL Shell Overview.docx > Samza SQL Shell > --- > > Key: SAMZA-1901 > URL: https://issues.apache.org/jira/browse/SAMZA-1901 > Project: Samza > Issue Type: New Feature > Components: sql >Reporter: Weiqing Yang >Priority: Major > Fix For: 0.15.0 > > Attachments: Samza SQL Shell Overview.docx > > > This Jira implements the first version of Samza sql shell. > Please refer to the attached document for more details about the shell, > including the tech choices, features, design decisions, how to build, run and > debug the shell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SAMZA-1901) Samza SQL Shell
Weiqing Yang created SAMZA-1901: --- Summary: Samza SQL Shell Key: SAMZA-1901 URL: https://issues.apache.org/jira/browse/SAMZA-1901 Project: Samza Issue Type: New Feature Components: sql Reporter: Weiqing Yang Fix For: 0.15.0 Attachments: Samza SQL Shell Overview.docx This Jira implements the first version of Samza sql shell. Please refer to the attached document for more details about the shell, including the tech choices, features, design decisions, how to build, run and debug the shell. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SAMZA-1821) Fetching systemStream configs causes duplicate key exception
[
https://issues.apache.org/jira/browse/SAMZA-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiqing Yang updated SAMZA-1821:
Fix Version/s: 1.0
> Fetching systemStream configs causes duplicate key exception
>
>
> Key: SAMZA-1821
> URL: https://issues.apache.org/jira/browse/SAMZA-1821
> Project: Samza
> Issue Type: Bug
> Components: sql
>Reporter: Weiqing Yang
>Priority: Major
> Fix For: 1.0
>
>
> If users input a samza.sql file including multiple Sql statements which have
> the same input sources or output destinations, for example,
> {code:java}
> insert into log.outputStream select * from
> brooklin.espresso.elasticsearchEnterpriseAccounts
> insert into log.outputstream select sfdcAccountId as key, organizationUrn as
> name2, description as name3 from
> brooklin.espresso.elasticsearchEnterpriseAccounts
> {code}
> an exception of "java.lang.IllegalStateException: Duplicate key
> org.apache.samza.sql.interfaces.SqlIOConfig" is thrown when fetching
> input/output systemStream configs:
> {code:java}
> java.lang.IllegalStateException: Duplicate key
> org.apache.samza.sql.interfaces.SqlIOConfig@1bf0f6f6
> at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133)
> at java.util.HashMap.merge(HashMap.java:1245)
> at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320)
> at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
> at
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
> at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.samza.sql.runner.SamzaSqlApplicationConfig.(SamzaSqlApplicationConfig.java:116)
> at
> com.linkedin.samza.sql.SamzaLiSqlParser.getSamzaSqlResources(SamzaLiSqlParser.java:58)
> at
> com.linkedin.samza.sql.SamzaLiSqlParser.parseSqlResources(SamzaLiSqlParser.java:51)
> at
> com.linkedin.samza.sql.SamzaLiSqlParserTest.testParseSqlResources_mixedKinds(SamzaLiSqlParserTest.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:661)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:869)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1193)
> at
> org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:126)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
> at org.testng.TestRunner.privateRun(TestRunner.java:744)
> at org.testng.TestRunner.run(TestRunner.java:602)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
> at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375)
> at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
> at org.testng.SuiteRunner.run(SuiteRunner.java:289)
> at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
> at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
> at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301)
> at org.testng.TestNG.runSuitesLocally(TestNG.java:1226)
> at org.testng.TestNG.runSuites(TestNG.java:1144)
> at org.testng.TestNG.run(TestNG.java:1115)
> at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
> at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
