Re: [VOTE] Apache Pinot (incubating) 0.7.0 RC0

2021-03-05 Thread Felix Cheung 
signing key should use a @apache.org address

- incubating in name
- signature and hash fine
- DISCLAIMER is fine
- LICENSE is fine

NOTICE should have year 2021?
Copyright 2018-2020 The Apache Software Foundation
https://www.apache.org/legal/src-headers.html#notice

- No unexpected binary files (some test .gz files)
- All source files have ASF headers



From: Yupeng Fu 
Sent: Friday, March 5, 2021 9:28:24 AM
To: dev@pinot.apache.org 
Subject: [VOTE] Apache Pinot (incubating) 0.7.0 RC0

Hi Pinot Community,

This is a call for vote to the release Apache Pinot (incubating) version
0.7.0.

The release candidate:
https://dist.apache.org/repos/dist/dev/incubator/pinot/apache-pinot-incubating-0.7.0-rc0/

Git tag for this release:
https://github.com/apache/incubator-pinot/tree/release-0.7.0-rc0

Git hash for this release:
157bd569303bbab60d79dbd36ba68089d1ee651f

The artifacts have been signed with key: F41AB2B856C6C29D, which can be
found in the following KEYS file.
https://dist.apache.org/repos/dist/release/incubator/pinot/KEYS

Release notes:
https://github.com/apache/incubator-pinot/releases/tag/release-0.7.0-rc0

Staging repository:
https://repository.apache.org/content/repositories/orgapachepinot-1023

Documentation on verifying a release candidate:
https://cwiki.apache.org/confluence/display/PINOT/Validating+a+release+candidate


The vote will be open for at least 72 hours or until the necessary number of
votes are reached.

Please vote accordingly,

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason

Thanks,
Apache Pinot (incubating) team

--
--Yupeng

Apache Pinot Daily Email Digest (2021-03-05)

2021-03-05 Thread Pinot Slack Email Digest 
#general@humengyuk18: Does Pinot support change schema existing column name? I tried change a column name, but got following exceptions on query:
```[
  {
"errorCode": 500,
"message": "MergeResponseError:\nData schema mismatch between merged block: [time_to_hour(LONG),age_decade(STRING),age_level(STRING),city(STRING),company_id(STRING),company_name(STRING),count_impression(LONG),count_in(LONG),count_passby(LONG),create_time(LONG),day(STRING),day_in_week(STRING),district(STRING),gate_id(STRING),gender(STRING),holiday_id(STRING),holiday_name(STRING),hour(STRING),is_holiday(STRING),month(STRING),province(STRING),region(STRING),shop_id(STRING),shop_name(STRING),temperature(STRING),temperature_id(STRING),total_duration(LONG),total_impression_duration(LONG),weather_cate_id(STRING),weather_cate_name(STRING),year(STRING)] and block to merge: [time_to_hour(LONG),age_decade(STRING),age_level(STRING),city(STRING),company_id(STRING),company_name(STRING),count_impression(LONG),count_in(LONG),count_passby(LONG),create_time(LONG),day(STRING),day_in_week(STRING),district(STRING),gate_id(STRING),gender(STRING),holiday_id(STRING),holiday_name(STRING),hour(STRING),is_holiday(STRING),month(STRING),province(STRING),region(STRING),shop_id(STRING),shop_name(STRING),temperature(STRING),temperature_id(STRING),total_duration(LONG),total_impression_duraion(LONG),weather_cate_id(STRING),weather_cate_name(STRING),year(STRING)], drop block to merge"
  }
]```  @mayanks: Hello, schema evolution is supported as long as it is backward compatible. Changing a column name or type is considered backward incompatible, and is not supported  @humengyuk18: Thanks, so in this case, I should delete all the segment and re-ingest all the data?  @mayanks: Yes, for incompatible schema change, that is the option @pankaj: If we extend a table schema in Pinot to add new columns (so it does not break backward compatibility); do we have to backfill data or can Pinot use null/default values to handle the older segments?  @mayanks: Pinot can auto fill null/default value in this case   @npawar: Pinot can also fill derived value i.e. if the value of new column is derived from existing columns, Pinot will calculate it using the function you provide@1705ayush: *How to ingest Data into pinot on kubernetes using native batch ingestion?*

Hi,

I am trying to ingest csv data into pinot deployed on kubernetes using LaunchDataIngestionJob arg.
I have verified that the table has been created on pinot and the job-spec and csv data are present on the node.
This is my job-spec file
```apiVersion: batch/v1
kind: Job
metadata:
  name: pinot-case-offline-ingestion
  namespace: my-pinot-kube
spec:
  template:
spec:
  containers:
- name: pinot-load-case-offline
  image: apachepinot/pinot:0.3.0-SNAPSHOT
  args: ["LaunchDataIngestionJob", "-jobSpecFile", "/opt/data/table-configs/case_history/job-spec.yml"]
  volumeMounts:
- name: mount-data
  mountPath: /opt/data
  restartPolicy: OnFailure
  volumes:
- name: mount-data
  hostPath:
path: /opt/data
  backoffLimit: 100```
After applying this job to node, nothing happens and this is the log of the pod.
```SegmentGenerationJobSpec: 
!!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec
excludeFileNamePattern: null
executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner,
  segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner,
  segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner}
includeFileNamePattern: glob:**/*.csv
inputDirURI: /opt/data/csv_data/case_prod_data
jobType: SegmentCreationAndTarPush
outputDirURI: /pinot-segments/case_history
overwriteOutput: true
pinotClusterSpecs:
- {controllerURI: ''}
pinotFSSpecs:
- {className: org.apache.pinot.spi.filesystem.LocalPinotFS, configs: null, scheme: file}
pushJobSpec: null
recordReaderSpec:
  className: org.apache.pinot.plugin.inputformat.csv.CSVRecordReader
  configClassName: org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig
  configs: {delimiter: '|', multiValueDelimiter: ''}
  dataFormat: csv
segmentNameGeneratorSpec:
  configs: {segment.name.prefix: case_history, exclude.sequence.id: 'true'}
  type: normalizedDate
tableSpec: {schemaURI: null, tableConfigURI: null, tableName: case_history}

Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS```
Am I ingesting the data incorrectly ?  @fx19880617: I think you are missing pushJobSpec?  @fx19880617: ```pushJobSpec: null```
  @1705ayush: Hi @fx19880617,
Thank you for helping.
I tried adding pushJobSpec to job-spec

Re: [VOTE] Apache Pinot (incubating) 0.7.0 RC0

2021-03-05 Thread kishore g 
+1



On Fri, Mar 5, 2021 at 11:09 AM Mayank Shrivastava 
wrote:

> +1
>
> Verified as per the steps listed.
>
> On Mar 5, 2021, at 9:28 AM, Yupeng Fu  wrote:
>
> 
> Hi Pinot Community,
>
> This is a call for vote to the release Apache Pinot (incubating) version
> 0.7.0.
>
> The release candidate:
>
> https://dist.apache.org/repos/dist/dev/incubator/pinot/apache-pinot-incubating-0.7.0-rc0/
>
> Git tag for this release:
> https://github.com/apache/incubator-pinot/tree/release-0.7.0-rc0
>
> Git hash for this release:
> 157bd569303bbab60d79dbd36ba68089d1ee651f
>
> The artifacts have been signed with key: F41AB2B856C6C29D, which can be
> found in the following KEYS file.
> https://dist.apache.org/repos/dist/release/incubator/pinot/KEYS
>
> Release notes:
> https://github.com/apache/incubator-pinot/releases/tag/release-0.7.0-rc0
>
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachepinot-1023
>
> Documentation on verifying a release candidate:
>
> https://cwiki.apache.org/confluence/display/PINOT/Validating+a+release+candidate
>
>
> The vote will be open for at least 72 hours or until the necessary number
> of
> votes are reached.
>
> Please vote accordingly,
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
> Thanks,
> Apache Pinot (incubating) team
>
> --
> --Yupeng
>
>

Re: [VOTE] Apache Pinot (incubating) 0.7.0 RC0

2021-03-05 Thread Mayank Shrivastava 
+1

Verified as per the steps listed.

> On Mar 5, 2021, at 9:28 AM, Yupeng Fu  wrote:
> 
> 
> Hi Pinot Community,
> 
> This is a call for vote to the release Apache Pinot (incubating) version
> 0.7.0.
> 
> The release candidate:
> https://dist.apache.org/repos/dist/dev/incubator/pinot/apache-pinot-incubating-0.7.0-rc0/
> 
> Git tag for this release:
> https://github.com/apache/incubator-pinot/tree/release-0.7.0-rc0
> 
> Git hash for this release:
> 157bd569303bbab60d79dbd36ba68089d1ee651f
> 
> The artifacts have been signed with key: F41AB2B856C6C29D, which can be
> found in the following KEYS file.
> https://dist.apache.org/repos/dist/release/incubator/pinot/KEYS
> 
> Release notes:
> https://github.com/apache/incubator-pinot/releases/tag/release-0.7.0-rc0
> 
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachepinot-1023
> 
> Documentation on verifying a release candidate:
> https://cwiki.apache.org/confluence/display/PINOT/Validating+a+release+candidate
> 
> 
> The vote will be open for at least 72 hours or until the necessary number of
> votes are reached.
> 
> Please vote accordingly,
> 
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> Thanks,
> Apache Pinot (incubating) team
> 
> -- 
> --Yupeng

[VOTE] Apache Pinot (incubating) 0.7.0 RC0

2021-03-05 Thread Yupeng Fu 
Hi Pinot Community,

This is a call for vote to the release Apache Pinot (incubating) version
0.7.0.

The release candidate:
https://dist.apache.org/repos/dist/dev/incubator/pinot/apache-pinot-incubating-0.7.0-rc0/

Git tag for this release:
https://github.com/apache/incubator-pinot/tree/release-0.7.0-rc0

Git hash for this release:
157bd569303bbab60d79dbd36ba68089d1ee651f

The artifacts have been signed with key: F41AB2B856C6C29D, which can be
found in the following KEYS file.
https://dist.apache.org/repos/dist/release/incubator/pinot/KEYS

Release notes:
https://github.com/apache/incubator-pinot/releases/tag/release-0.7.0-rc0

Staging repository:
https://repository.apache.org/content/repositories/orgapachepinot-1023

Documentation on verifying a release candidate:
https://cwiki.apache.org/confluence/display/PINOT/Validating+a+release+candidate


The vote will be open for at least 72 hours or until the necessary number of
votes are reached.

Please vote accordingly,

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason

Thanks,
Apache Pinot (incubating) team

-- 
--Yupeng